Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

This paper claims to deliver the largest publicly available sentence-level legal argument mining corpus for U.S. caselaw: 2,900,083 sentences from U.S. Supreme Court opinions (1921–2025), each automatically labeled as Fact, Issue, Rule/Law/

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LAMUS adds a roughly 2.9-million-sentence LLM-labeled corpus of U.S. Supreme Court opinions to legal argument mining, with a smaller human-verified Texas benchmark.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A potentially useful 2.9M-sentence SCOTUS argument-mining resource, but the quality evidence rests on a thin 600-sentence check and a never-evaluated annotation model. the 5 major comments →

arxiv 2603.08286 v2 pith:XIIJQ2LP submitted 2026-03-09 cs.CL

LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs

Serene Wang , Lavanya Pobbathi , Haihua Chen This is my paper
classification cs.CL
keywords legal argument miningcorpus constructionSupreme Court caselawlarge language modelssentence classificationchain-of-thought promptingfine-tuningannotation quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims to provide the largest publicly available sentence-level legal argument mining resource for U.S. caselaw: 2,900,083 sentences from Supreme Court opinions (1921–2025), each labeled as Fact, Issue, Rule/Law/Holding, Analysis, Conclusion, or Other. The labels were produced automatically by a 70-billion-parameter language model chosen after benchmarking on a small human-annotated Texas criminal appeal corpus, then verified by a hybrid human-in-the-loop step. The paper also claims systematic experiments show that fine-tuning an 8-billion-parameter model reaches 85.32% accuracy, chain-of-thought prompting helps capable models, and few-shot prompting with generic examples hurts. A sympathetic reader would care because this fills a gap in U.S. legal NLP resources and offers practical guidance on how to build and use legal argument classifiers at scale.

Core claim

The central claim is that U.S. judicial opinion text can be automatically labeled at scale into the six functional components of legal argument, and that the resulting corpus is reliable enough to support training and evaluation. The authors describe a data-centric pipeline — collect opinions, segment sentences, run an LLM annotator, flag inconsistencies with a second LLM, and have humans review flagged items — and report that human verification on a stratified 600-sentence sample reached Cohen's κ = 0.85 with 89.2% average agreement with the model. They additionally claim that fine-tuned models outperform prompting (best accuracy 85.32%) and that the corpus reveals a pronounced domain shift

What carries the argument

The operational core is a six-class sentence-level annotation scheme grounded in the FIRAC framework (Fact, Issue, Rule/Law/Holding, Analysis, Conclusion, Other). The carrying mechanism is a pipeline that uses a large language model for bulk annotation, a second LLM to flag likely mislabels, and targeted human review to correct them. Model selection and prompt evaluation were run on a 4,042-sentence human-annotated Texas corpus; fine-tuning experiments used 2,585 labeled sentences with QLoRA. The paper's key empirical contrasts are zero-shot vs few-shot vs chain-of-thought prompting, and prompting vs fine-tuning.

Load-bearing premise

The load-bearing premise is that a model selected and tuned on a small Texas criminal-appeal corpus, where Facts dominate at 61.9%, will label accurately on a very different Supreme Court corpus, where Rule/Law/Holding and Analysis dominate at 27.6% each — a transfer the paper asserts but never validates on the full 2.9M labels.

What would settle it

Take a stratified random sample of, say, 1,500 Supreme Court sentences across court eras, have two independent legal experts label them, and compare against the corpus labels. If human-model agreement falls well below the reported 89.2% average, or if the estimated error rate exceeds the 19.4% pre-correction error reported for the Texas set, the transfer assumption fails. A simpler check: run the actual 70B annotator on a human-labeled SCOTUS gold standard, since the paper reports no direct evaluation of that model.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If LAMUS is reliable, U.S. legal NLP gains a large federal-corpus complement to the existing non-U.S. argument mining datasets, enabling training and evaluation on American judicial reasoning.
  • The recommended deployment strategy — fine-tune when accuracy matters, use chain-of-thought prompting when fine-tuning is unavailable — follows directly from the accuracy gap (85.32% vs 75.89%).
  • The monotone few-shot degradation (67.23% with zero examples down to 53.94% with 100) implies that generic demonstration examples should not be used for jurisdiction-specific legal classification.
  • The era-by-era label distributions (e.g., Burger Court largest at 809,409 sentences) provide a basis for diachronic studies of how argumentation style changes across courts.
  • The documented domain shift (Facts 61.9% → 26.3%; Rule/Law/Holding 5.3% → 27.6%) implies that models trained on one court type will need adaptation before transferring to another.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because human verification covered only 600 of 2.9M sentences and sampled 100 per category, the corpus-level error rate is largely an extrapolation; a natural extension would estimate per-era and per-category error rates on a larger stratified sample.
  • The few-shot failure may be an artifact of using Texas criminal examples for a Supreme Court target; testing few-shot with in-domain SCOTUS exemplars would determine whether the negative result is about domain mismatch or few-shot learning itself.
  • The same LLM-annotate-plus-judge-plus-human pipeline could be exported to other low-resource legal genres (state civil opinions, administrative decisions, international tribunals) with the Texas corpus serving only as a development set.
  • The paper's label distribution data suggests that argument-role priors are jurisdiction-specific; downstream models should either calibrate class priors per corpus or be trained on distribution-matched data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces LAMUS, a sentence-level legal argument mining corpus built from U.S. Supreme Court (SCOTUS) opinions and Texas criminal appellate opinions. The authors define a six-class annotation scheme (Fact, Issue, Rule/Law/Holding, Analysis, Conclusion, Other), construct a refined 4,042-sentence Texas seed corpus, evaluate several LLMs and LegalBERT under zero-shot, few-shot, chain-of-thought, and fine-tuning settings, and then use LLaMA-3-70B to automatically label 2,900,083 sentences from SCOTUS opinions spanning 1921–2025. They report that fine-tuning LLaMA-3-8B reaches 85.32% accuracy, that few-shot prompting consistently degrades performance, and that a 600-sentence human verification yields Cohen's kappa 0.85 with 89.2% average human-model agreement. The paper argues that LAMUS is the largest publicly available sentence-level U.S. legal argument mining resource.

Significance. If the corpus-quality claims hold, LAMUS would be a substantial contribution to legal NLP: a 2.9M-sentence, six-class argument mining resource with temporal coverage of nearly a century of SCOTUS opinions, released with code and data links. The paper also provides useful empirical findings: chain-of-thought prompting helps larger general-purpose models, few-shot prompting with fixed generic examples hurts performance, and fine-tuning dominates prompting for this classification task. The 600-sentence human verification, the explicit domain-shift analysis (Table 17), and the 36-run ablation study are positive methodological features. However, the central claim that the 2.9M automatically assigned labels are high quality rests on evidence that is currently too thin, partly circular, and partly inconsistent with the reported numbers. These gaps are fixable and do not invalidate the resource concept, but they block acceptance as stated.

major comments (5)
  1. [§4.1.3, §5.2] The final LAMUS corpus is annotated with LLaMA-3-70B, but no evaluation of LLaMA-3-70B appears anywhere in the paper. Table 3's largest evaluated model is SaulLM-54B, and the best prompted model is LLaMA-3-8B with CoT at 75.89%. The statement in §5.2 that 70B 'demonstrated the highest performance in our preliminary evaluations' is unsupported by any reported number. Moreover, §5.2 describes the model as identifying 'claims, premises, and supporting or opposing reasoning,' which is a different task than the six-class scheme used elsewhere. The exact prompt used for the 2.9M corpus is not given. This is load-bearing because the reader cannot verify the competence of the model that actually produced the corpus. Please report the 70B evaluation on the same benchmark, specify the exact annotation prompt, or use a model whose performance is documented.
  2. [§5.3, Table 17] The 600-sentence human verification samples 100 sentences per category, i.e., a balanced sample, while the actual SCOTUS distribution in Table 17 is strongly skewed (Facts 26.3%, Rule/Law/Holding 27.6%, Analysis 27.6%, Others 12.2%, Conclusion 4.2%, Issue 2.1%). The aggregate 89.2% human-model agreement is not weighted by this distribution, and per-category agreement is not reported. If the model agrees less with humans on rare categories such as Issue or Conclusion, the corpus-level accuracy on the true label mix could be materially lower than 89.2%. Please provide a per-category agreement table and the distribution-weighted agreement under the Table 17 proportions.
  3. [§4.1.2] The claimed 19.4% error-correction rate is not supported by the presented numbers. GPT flagged 1,058 of 4,042 sentences; manual review confirmed 273 as incorrect. 273/4,042 = 6.8%, not 19.4%. The additional estimate of 785 total incorrect labels depends on 'targeted manual spot checks' whose sample size and error-counting procedure are not specified. Without the estimation formula, the 'nearly one-fifth' claim in the abstract and §3.3 is unverifiable. Either report the precise calculation or state only the confirmed 273 corrections.
  4. [§5.3, §3.3, Table 1] The quality validation is partly circular. The SCOTUS human annotators were given the same Table 1 category definitions and guidelines that generated the LLaMA labels, so human-model agreement partly measures whether the prompt instructions were followed, not whether the labels are correct in an external sense. In addition, the Texas 'gold' labels used for model selection were themselves revised after GPT-4 flagging (§4.1.2). An independent evaluation—for example, adjudication by annotators who do not see the LLM labels, or a comparison against an external legal-annotation benchmark—is needed to certify label quality for the 2.9M sentences.
  5. [§A.6, §5.2] Table 17 documents a large domain shift between the Texas training corpus and the SCOTUS target corpus (Facts −35.6 points, Rule/Law/Holding +22.2 points, Analysis +13.3 points). Selecting LLaMA-3-70B based on Texas criminal appeal data is therefore not by itself evidence of SCOTUS performance. The only SCOTUS-specific validation is the 600-sentence check in §5.3, but that sample is balanced rather than distribution-matched and is only 0.02% of the corpus. To support the central corpus-quality claim, the authors should either add a larger, distribution-proportional SCOTUS evaluation or substantially temper the claim that the full 2.9M-label corpus has verified annotation quality.
minor comments (5)
  1. [§6.2, §A.2] Future Work says 'One promising direction is to conduct stability testing,' but §A.2 already reports a 10-run stability test with LLaMA-3-8B CoT. This appears to be an internal inconsistency; remove or rephrase the Future Work sentence.
  2. [§4.3.5, Table 8, Table 12, Figure 4] The best configuration is reported inconsistently: §4.3.5 says 'learning rate 2e-4, 3 epochs, LoRA rank 16,' Table 8/Table 12 and Figure 4 indicate LR=1e-4, Epochs=5, Rank=8 (or 16). Please reconcile the text, tables, and figure caption.
  3. [§5.1, §4.1.3, Table 11] The corpus is described as spanning 'from 1759 to the present' in §5.1, but §4.1.3 and Table 11 state 1921–2025. Clarify which date range applies to the released LAMUS corpus.
  4. [Table 4, Table 5] The zero-shot baseline differs between Table 4 (65.38%) and Table 5 (67.23%) due to prompt formatting. The footnote explains this, but the explanation should appear near the first table to avoid confusing readers.
  5. [§4.2] The fine-tuning description says models were trained on 2,585 labeled sentences from the corpus, while §4.1.2 says the enhanced Texas dataset contains 4,042 sentences. Please clarify the relationship (e.g., train/validation split or a subset).

Circularity Check

0 steps flagged

No significant circularity: the corpus pipeline is an empirical construction with independent human verification; the main concerns are missing LLaMA-3-70B evaluation and a small stratified human sample, not circular reasoning.

full rationale

The claimed derivation chain is linear and observational. The Texas seed corpus and six-class scheme are taken from the authors' prior work (Chen et al. [7]); this is a self-citation, but it supplies data and definitions that are externally published and available, and the final SCOTUS labels are not derived from that corpus by construction. The model-selection step is empirical: models are benchmarked on the refined Texas set, and the selected model (LLaMA-3-70B) is then applied to SCOTUS. The only quality validation of the 2.9M labels is the human verification in Sec. 5.3 (600 sentences, Cohen's kappa 0.85, human-model agreement 89.2%). Using the same six category definitions for both LLM prompting and human annotators is the standard annotation protocol, not a reduction of the evaluation to the model's own outputs; annotators labeled independently without discussion, and their labels are external to the LLM's predictions. The GPT-4 verification in Sec. 3.3 flags candidate mislabels that are then adjudicated by humans, so the 'nearly 20% correction' is not GPT-4 relabeling the gold standard by itself. No fitted parameter is renamed as a prediction and no uniqueness theorem is imported. The paper's genuine weaknesses are non-circular: Sec. 5.2 asserts LLaMA-3-70B 'demonstrated the highest performance in our preliminary evaluations' but no 70B evaluation is reported; the 600-sentence sample is stratified 100 per model label rather than proportional to the Table 17 distribution; and Sec. A.6 documents a large domain shift (Facts 61.9%->26.3%, Rule/Law/Holding 5.3%->27.6%). These affect confidence in corpus quality but do not make the derivation equivalent to its inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper contains no theoretical derivation; its central contribution is a dataset. The ledger therefore lists the annotation-scheme, transfer, and quality-assurance assumptions that the corpus's value depends on.

free parameters (3)
  • GPT verification flagging threshold = not specified
    The step that flags sentences for manual review depends on a confidence threshold that is never stated; it controls which 1,058 sentences were reviewed and therefore shapes the correction estimate.
  • Fine-tuning hyperparameters (LR, LoRA rank, epochs) = best: LR=1e-4, rank=8, epochs=5 (Table 12; text conflicts in §4.3.5)
    Selected by grid search on the Texas test set; reported as final accuracy, so it is a fitted value rather than a held-out prediction.
  • Few-shot exemplar set = 100 curated sentences in 20 groups of 5
    Hand-constructed; the negative few-shot result depends on these specific examples, not on few-shot prompting in general.
axioms (4)
  • domain assumption Sentence-level classification is a faithful representation of legal argument structure
    The entire task is defined as assigning one FIRAC label per sentence; long-range structure and multi-label sentences are excluded (Section 3.1).
  • domain assumption The FIRAC six-class scheme is the correct annotation ontology
    Adopted from Chen et al. [7]; no external validation that these categories capture judicial reasoning.
  • domain assumption LLM labels are reliable enough to serve as corpus annotations
    The final SCOTUS corpus is machine-labeled; human verification covers only 600 sentences and does not correct the remaining 2.9M (Sections 5.2–5.3).
  • ad hoc to paper Model performance on Texas criminal appeals transfers to SCOTUS opinions
    The annotation model is selected using Texas data, but Table 17 shows substantial distribution shift; no SCOTUS fine-tuning is reported.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs." pith.science (2026). https://pith.science/paper/XIIJQ2LP

@misc{pith2026260308286,
  author       = {Pith},
  title        = {Pith review of: LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XIIJQ2LP}},
  note         = {Machine review of arXiv:2603.08286}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Legal argument mining aims to identify and classify the functional components of judicial reasoning, such as facts, issues, rules, analysis, and conclusions. Progress in this area is limited by the lack of large-scale, high-quality annotated datasets for U.S. caselaw, particularly at the state level. This paper introduces LAMUS, a sentence-level legal argument mining corpus constructed from U.S. Supreme Court decisions and Texas criminal appellate opinions. The dataset is created using a data-centric pipeline that combines large-scale case collection, LLM-based automatic annotation, and targeted human-in-the-loop quality refinement. We formulate legal argument mining as a six-class sentence classification task and evaluate multiple general-purpose and legal-domain language models under zero-shot, few-shot, and chain-of-thought prompting strategies, with LegalBERT as a supervised baseline. Results show that chain-of-thought prompting substantially improves LLM performance, while domain-specific models exhibit more stable zero-shot behavior. LLM-assisted verification corrects nearly 20% of annotation errors, improving label consistency. Human verification achieves Cohen's Kappa of 0.85, confirming annotation quality. LAMUS provides a scalable resource and empirical insights for future legal NLP research. All code and datasets can be accessed for reproducibility on GitHub at: https://github.com/LavanyaPobbathi/LAMUS/tree/main

Figures

Figures reproduced from arXiv: 2603.08286 by Haihua Chen, Lavanya Pobbathi, Serene Wang.

Figure 1
Figure 1. Figure 1: Methodology for corpus construction, LLM-based [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of Texas Case Law dataset sentence [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy of LLaMA-3-8B and SaulLM-54B un [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation Study: Learning Rate vs. Accuracy by LoRA Rank and Training Epochs. Results from 36 experiments across [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overall distribution of legal argument label cate [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of legal argument label distributions [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Overall comparison of experimental results across [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HKJudge: A Legal Discourse-Annotated Corpus for Interpreting What Courts Find, How They Reason, and What They Rule

    cs.CL 2026-06 accept novelty 8.0

    HKJudge is a new ~290k-sentence expert-annotated corpus of Hong Kong criminal judgments with 26 rhetorical roles and 3 sentencing elements, plus benchmarks on classification and extraction tasks.

  2. Structural Dilemmas and Developmental Pathways of Legal Argument Mining in the Era of Artificial Intelligence

    cs.CL 2026-05 unverdicted novelty 3.0

    Legal argument mining advances slowly because no structured way exists to represent arguments that balances rich legal theory with what computers can process, creating dilemmas in data standardization, modeling, and d...

Reference graph

Works this paper leans on

42 extracted references · 6 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Alibaba Cloud. 2025. Qwen3 Technical Report. https://qwenlm.github.io/. Reasoning-oriented large language models

    2025

  2. [2]

    Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language pro- cessing for the legal domain: A survey of tasks, datasets, models, and challenges. Comput. Surveys58, 6 (2025), 1–37

    2025

  3. [3]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. InProceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623

    2021

  4. [4]

    Michael J Bommarito II, Daniel Martin Katz, and Eric M Detterman. 2021. LexNLP: Natural language processing and information extraction for legal and regulatory texts. InResearch handbook on big data law. Edward Elgar Publishing, 216–227

    2021

  5. [5]

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The Muppets straight out of Law School. InFindings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 2898–2904. https://doi.org/1...

    doi:10.18653/v1/2020.findings-emnlp 2020

  6. [6]

    Haihua Chen, Jiangping Chen, and Junhua Ding. 2021. Data evaluation and enhancement for quality improvement of machine learning.IEEE Transactions on Reliability70, 2 (2021), 831–847

    2021

  7. [7]

    Haihua Chen, Lavinia F Pieptea, and Junhua Ding. 2022. Construction and evaluation of a high-quality corpus for legal intelligence using semiautomated approaches.IEEE Transactions on Reliability71, 2 (2022), 657–673

    2022

  8. [8]

    Xiao Chi, Wei Wang, Ziyao Zhang, Ang Li, Yuting Huang, Yiquan Wu, Kun Kuang, Changlong Sun, Xiaozhong Liu, Fei Wu, et al. 2026. LegalAi research in LLM Era: data, modeling and evaluation.Artificial Intelligence Review(2026)

    2026

  9. [9]

    Eunjung Cho, Alexander Miserlis Hoyle, and Yoan Hermstrüwer. 2025. Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarization. InProceedings of the Natural Legal Language Processing Workshop

    2025

  10. [10]

    Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. 2024. Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain. Advances in Neural Information Processing Systems37 (2024), 129672–129695

    2024

  11. [11]

    Rajaa El Hamdani, Thomas Bonald, Fragkiskos D Malliaros, Nils Holzenberger, and Fabian Suchanek. 2024. The factuality of large language models in the legal domain. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management. 3741–3746

    2024

  12. [12]

    Joseph Enguehard, Morgane Van Ermengem, Kate Atkinson, Sujeong Cha, Ari- jit Ghosh Chowdhury, Prashanth Kallur Ramaswamy, Jeremy Roghair, Hannah R Marlowe, Carina Suzana Negreanu, Kitty Boxall, et al. 2025. LeMAJ (Legal LLM- as-a-Judge): Bridging Legal Reasoning and LLM Evaluation. InProceedings of the Natural Legal Language Processing Workshop 2025. 318–337

    2025

  13. [13]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences120, 30 (2023), e2305016120

    2023

  14. [14]

    Google DeepMind. 2025. Gemini Model Family. https://ai.google.dev/gemini-api. Including Gemini 2.5 Flash

    2025

  15. [15]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zam- brano, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.Advances in neural information pro- cessing systems36 (2023), 44123–44279

    2023

  16. [16]

    Xue Guo, Yuting Huang, Bin Wei, Kun Kuang, Yiquan Wu, Leilei Gan, Xianshan Huang, and Xianglin Dong. 2025. Specialized or general AI? a comparative evaluation of LLMs’ performance in legal tasks.Artificial Intelligence and Law (2025), 1–37

    2025

  17. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300(2020)

    Pith/arXiv arXiv 2020

  18. [18]

    Yiran Hu, Huanghai Liu, Chong Wang, Kunran Li, Tien-Hsuan Wu, Haitao Li, Xinran Xu, Siqing Huo, Weihang Su, Ning Zheng, et al . 2026. Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions.arXiv preprint arXiv:2601.15267(2026)

    arXiv 2026

  19. [19]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  20. [20]

    Manuj Kant, Sareh Nabi, Manav Kant, Roland Scharrer, Megan Ma, and Marzieh Nabi. 2025. Towards robust legal reasoning: Harnessing logical llms in law.arXiv preprint arXiv:2502.17638(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners.Advances in neural information processing systems35 (2022), 22199–22213

    2022

  22. [22]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.biometrics(1977), 159–174

    1977

  23. [23]

    Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. 2025. Legalagentbench: Evaluating llm agents in legal domain. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2322–2344

    2025

  24. [24]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods.arXiv preprint arXiv:2412.05579(2024)

    Pith/arXiv arXiv 2024

  25. [25]

    Davide Liga and Livio Robaldo. 2023. Fine-tuning GPT-3 for legal rule classifica- tion.Computer Law & Security Review51 (2023), 105864

    2023

  26. [26]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.ACM computing surveys55, 9 (2023), 1–35

    2023

  27. [27]

    Meta AI. 2024. The Llama 3 Model Family. https://ai.meta.com/llama/

    2024

  28. [28]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work?arXiv preprint arXiv:2202.12837(2022)

    Pith/arXiv arXiv 2022

  29. [29]

    Raquel Mochales and Marie-Francine Moens. 2011. Argumentation mining. Artificial intelligence and law19, 1 (2011), 1–22

    2011

  30. [30]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (2022), 27730–27744

    2022

  31. [31]

    Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: the detection, classification and structure of arguments in text. InProceedings of the 12th international conference on artificial intelligence and law. 98–107

    2009

  32. [32]

    Prakash Poudyal, Jaromír Šavelka, Aagje Ieven, Marie Francine Moens, Teresa Goncalves, and Paulo Quaresma. 2020. ECHR: Legal corpus for argument mining. InProceedings of the 7th Workshop on Argument Mining. 67–75

    2020

  33. [33]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.OpenAI blog 1, 8 (2019), 9

    2019

  34. [34]

    Peizhang Shao, Linrui Xu, Jinxi Wang, Wei Zhou, and Xingyu Wu. 2025. When large language models meet law: Dual-lens taxonomy, technical advances, and ethical governance.arXiv preprint arXiv:2507.07748(2025)

    Pith/arXiv arXiv 2025

  35. [35]

    Dong Shu, Haoran Zhao, Xukun Liu, David Demeter, Mengnan Du, and Yongfeng Zhang. 2024. Lawllm: Law large language model for the us legal system. InPro- ceedings of the 33rd ACM International Conference on information and knowledge management. 4882–4889

    2024

  36. [36]

    Marco Siino, Mariana Falco, Daniele Croce, and Paolo Rosso. 2025. Exploring llms applications in law: A literature review on current legal nlp approaches. IEEE Access13 (2025), 18253–18276

    2025

  37. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  38. [38]

    Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction.arXiv preprint arXiv:1807.02478 (2018)

    Pith/arXiv arXiv 2018

  39. [39]

    Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. 2025. Data-centric artificial intelligence: A survey. Comput. Surveys57, 5 (2025), 1–42

    2025

  40. [40]

    Gechuan Zhang, Paul Nulty, and David Lillis. 2022. Enhancing legal argument mining with domain pre-training and neural networks.Journal of Data Mining & Digital Humanities(2022)

    2022

  41. [41]

    Li Zhang, Matthias Grabmair, Morgan Gray, and Kevin Ashley. 2025. Thinking Longer, Not Always Smarter: Evaluating LLM Capabilities in Hierarchical Legal Reasoning.arXiv preprint arXiv:2510.08710(2025). arXiv, 2026, Wang, Pobbathi and Chen

    arXiv 2025

  42. [42]

    Facts, ” the majority class, with F1-scores exceeding 0.82. Performance drops substantially for mi- nority classes: “Analysis

    Dan Zhou, Zhijing Tang, Huan Sun, Weizhe Chen, et al . 2023. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. InICLR. A Appendix This appendix provides comprehensive supplementary results from our experimental evaluation. We present the complete ablation grid covering all 36 hyperparameter configurations, stability test results ...

    2023

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.