Pith. sign in

REVIEW 3 major objections 4 minor 39 references

LLMs for Legal Subsumption in German Employment Contracts

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLMs reach 80% weighted F1 and high void-clause recall on German employment contract clauses when given lawyer-written examination guidelines, while full-text legal sources still leave them below human lawyers.

desk verdict Solid, reproducible dataset work with a transparent evaluation, but the headline guideline-result is weakened by the guidelines being derived from the same clauses used for testing. read the letter →

arxiv 2507.01734 v1 pith:YMRHPDO4 submitted 2025-07-02 cs.CL

Oliver Wardas , Florian Matthes This is my paper
classification cs.CL
keywords legalsubsumptionGermanemploymentcontractsexaminationguidelineslargelanguagemodelsin-contextlearningcontractlegalityclassificationvoidclausedetectionNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Legal subsumption—deciding whether a particular contractual clause is void under the applicable law—is exactly the kind of text-heavy, high-stakes task that could benefit from language models. This paper tests whether large language models can perform that judgment on clauses from German employment contracts, comparing three settings: no legal context, the full text of laws and court rulings, and distilled rules written by lawyers (examination guidelines). The central result is that the lawyer-written guidelines outperform full-text sources, raising the best model's weighted F1 to 80% and recall for void clauses to 80% or higher for several models. With full-text sources, performance improves only moderately and remains substantially below that of human lawyers. The authors conclude that LLMs can usefully assist lawyers when legal knowledge is pre-digested, but cannot yet subsume directly under original legal sources.

What carries the argument

The central object is the examination guideline: a short, distilled legal rule written by human lawyers that states the condition under which a group of clauses is void and lists the laws and court rulings from which the rule derives. In the guideline variant of the experiment, all 24 guidelines are placed in the system prompt, so the model must select applicable rules rather than retrieve them; in the full-text variant, original sources are supplied as if by perfect retrieval. The guidelines carry the argument because the gap between guideline-based and source-based performance is the paper's main evidence about where LLM legal subsumption currently fails.

What would settle it

An independent evaluation on newly drafted employment clauses never seen by the guideline authors, with guidelines derived solely from statutes and court rulings, would settle whether the 80% weighted F1 reflects general subsumption ability; if void recall collapses to baseline on such held-out clauses, the guideline-based result is an artifact of rules reverse-engineered from the test set.

Watch

Extended reading notes

Core claim

The paper's own claim is that the form in which legal knowledge is presented, not just model capability, determines whether LLMs can perform legal subsumption. Using 24 examination guidelines distilled from 41 laws and court rulings, the strongest model reaches an 80% weighted F1-score and 80% recall on void clauses, and several other models achieve void recall above 80% (up to 98% for the reasoning model DeepSeek-R1, albeit with lower precision). When the same models receive the original full-text sources, their performance falls, and the paper reads this gap as evidence that state-of-the-art LLMs can apply pre-digested legal rules but lag human lawyers at deriving subsumptions from primary legal materials. The paper also finds that every model improves when any legal context is supplied, which it takes as support for retrieval-augmented and knowledge-base approaches to legal NLP.

Load-bearing premise

The load-bearing premise is that the examination guidelines, written by lawyers who grouped the dataset's own void clauses, serve as an independent source of legal knowledge rather than as a description of the test set.

Editorial extensions

If this is right

  • If the guidelines-based result holds, a semi-automated contract review workflow that maintains a human-curated legal knowledge base and uses LLMs to produce initial drafts is within reach.
  • High void-clause recall—80% for the leading model, 82% for another, 98% for the reasoning model—attacks the most costly failure mode, because missing a void clause is the risk lawyers most need to avoid.
  • Because full-text sources only moderately improve predictions, applications built directly on raw laws and court rulings cannot yet be trusted for unassisted review, even under a perfect retrieval assumption.
  • The consistently poor performance on the unfair class means the models cannot yet separate unfair-but-valid clauses from void ones, so human oversight remains essential.
  • Providing any legal context improves every model's averaged F1 over internal knowledge alone, supporting retrieval-augmented and knowledge-base approaches to legal NLP.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because the examination guidelines were distilled from the same clauses used in the test set, the high void recall may partly reflect rules tailored to those clauses; re-running with guidelines written only from statutes and rulings, or with newly drafted held-out clauses, would reveal how much general subsumption ability remains.
  • My inference: the guideline/full-text gap may be driven by input length and salience rather than legal reasoning alone; benchmarking with short extractive summaries of the same sources, or with guidelines of comparable length to source excerpts, would isolate the cause.
  • My inference: DeepSeek-R1's near-perfect void recall with low precision and a general bias toward void classifications suggests a natural human-in-the-loop design: use a high-recall model to flag candidate clauses and a higher-precision model to prepare the final draft.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper extends an existing German employment contract clause dataset with lawyer-created examination guidelines, references to laws and court rulings, and full-text legal sources, then evaluates five LLMs on a three-way legality classification task (valid/unfair/void) under three context variants: no legal context, full-text legal sources, and distilled examination guidelines. The main empirical claim is that providing examination guidelines yields a weighted F1 of about 80% and high recall for void clauses, while full-text sources improve over no context but remain below the performance attributed to human lawyers. The authors release the extended dataset, code, and all log files. The paper positions the guideline condition as evidence that LLMs can assist lawyers in contract legality review, while the full-text condition is used to argue that LLMs still lag behind humans in direct legal subsumption from original sources.

Significance. If the reported results were robust and generalizable, the paper would make a useful contribution by demonstrating a concrete human-in-the-loop setup for German employment contract review and by releasing a valuable resource: 24 examination guidelines, 41 referenced legal sources, and annotations linking void clauses to guidelines. The release of code and log files is a genuine strength and supports reproducibility of the experiments as run. However, the significance of the central claim is materially limited by the evaluation design: the examination guidelines were constructed from the same void clauses that constitute the test set, so the guideline condition may measure rule-matching against a test-set-derived rule base rather than generalizable legal subsumption. The comparison with human lawyers is also asserted without a reported human-subject experiment. These issues mean the paper currently establishes an interesting experimental comparison but not the broader capability claim stated in the abstract.

major comments (3)
  1. [§4.1, §5.4, Table 3] The 24 examination guidelines were created by having lawyers group the dataset's own 112 void clauses and formulate one rule per group (Section 4.1). Evaluating those same clauses with those guidelines in the prompt therefore measures the model's ability to match clause text to a rule abstracted from that same text, not subsumption under an independently formulated body of legal knowledge. The reported 80% weighted F1 and void recall values up to 98% (Table 3) are consistent with reverse-engineering the test set and do not establish generalizable subsumption ability. I recommend a held-out evaluation in which guidelines are derived only from training clauses and applied to unseen clauses, or at minimum a per-guideline analysis reporting how many void clauses are near-paraphrases of the guideline text assigned to them.
  2. [§5.3, §5.4] The full-text variant uses gold annotations to supply exactly the references under which each void clause is void, while for non-void clauses it randomly selects two topic-matched sources. This is an oracle retrieval setup, which is acceptable for isolating subsumption, but it is not a neutral comparison against the guideline condition: both informative conditions derive their content from the same lawyer-constructed mapping from test clauses to legal rules. In addition, the instruction to classify a clause as void only if it can be subsumed under the given sources, combined with arbitrary source selection for valid and unfair clauses, may push predictions toward the majority 'valid' class. Please report full per-class confusion matrices for every condition and discuss how the oracle source selection affects the comparison between the guideline and full-text variants.
  3. [§5.4, §6.1] The paper repeatedly states that LLM performance with full-text legal sources is 'substantially below that of human lawyers,' but no human-subject experiment on this dataset or under the same evaluation protocol is reported. This claim is load-bearing for the conclusion that LLMs cannot yet perform unassisted legality review from original sources. The authors should add a human baseline measured on the same clauses and context variants, or reframe the claim as a comparison with expert-derived guidelines rather than with human performance.
minor comments (4)
  1. [§5.4] There are typos in the prose: 'lack behind' should be 'lag behind' in General Observations, and 'hightest' should be 'highest' in the GPT-4o Performance paragraph.
  2. [Table 3, §5.2] No confidence intervals or repeated-run variance are reported even though API-based LLM outputs are stochastic and the seed parameter is not supported for DeepSeek-V3 and DeepSeek-R1. A brief statement about single-run results and their reproducibility would strengthen the empirical claims.
  3. [§5.2] The 'hurt_rules' output field is only collected in the guideline condition, but the paper does not analyze whether the models' stated rule IDs match the gold guideline annotations. An analysis of these outputs would directly address whether high void recall is accompanied by correct identification of the applicable guideline.
  4. [§5.4] The missing DeepSeek-R1 cell in the full-text condition is explained, but Table 3 would benefit from a footnote summarizing how the uneven model-by-condition design affects cross-model comparisons.

Circularity Check

2 steps flagged · score 6.0 of 10

Examination guidelines were reverse-engineered from the same void clauses used for evaluation (Sec. 4.1), so the reported 80% F1 and void recall measure matching against a test-set-derived rule base, not generalizable legal subsumption.

  1. fitted input called prediction [Section 4.1 (Annotation Process), Section 5.2 (Setup), Section 5.4 (Results, Table 3)]
    "The examination guidelines were created by two lawyers from a German law firm, specialized in economic law. First, they grouped clauses together which were void for the same reason. Then they formulated one examination guideline for each group, describing the legal argument under which these clauses would be ruled void in court."

    The 24 examination guidelines are constructed by grouping the dataset's own 112 void clauses: lawyers grouped clauses void for the same reason and wrote one guideline per group. Section 4 adds that every void clause is annotated with the guideline under which it can be subsumed as void. Section 5.2 then places all guidelines in the prompt for every clause. So for each void test item, the prompt contains the very rule that was distilled from that item's group. The high void recall (80-98%) and 80% weighted F1 therefore measure how well a model can match a clause to rule text reverse-engineered from that clause's own label group; it does not measure whether the guidelines generalize to unseen contracts.

  2. other [Section 5.3 (Full-Text Sources)]
    "In this variant, the court rulings and laws provided to the models are filtered to simulate a perfect retrieval system. This ensures that if a clause is void, the sources include the exact references under which it can be subsumed as void. To achieve this, we utilize the annotations linking void clauses to examination guidelines, which in turn reference the relevant sources."

    For void clauses, the set of sources given to the model is selected from the gold annotation chain clause -> guideline -> sources. Thus the input legal context is a function of the true label. The prompt instructs the model to classify a clause as void only if its invalidity can be derived from the given sources, so the correct answer's legal basis is guaranteed to be among the documents. This turns the prediction into an oracle-supported entailment task. The authors disclose this as a 'perfect retrieval' assumption and do not claim end-to-end retrieval; nevertheless, the reported full-text improvement is an upper bound that already contains the answer-relevant sources, rather than a fully independent predictive result.

full rationale

The paper's headline positive result is that examination guidelines yield an 80% weighted F1 and high recall for void clauses (Table 3). The guidelines are not independent of the evaluation set: Section 4.1 says lawyers grouped the dataset's own void clauses by reason and formulated one guideline per group, and Section 4 says every void clause is annotated with the guideline under which it is subsumable. Section 5.2 feeds all guidelines into the prompt for every clause. Consequently, for each void test item the prompt contains the distilled rule that was itself derived from that item's group. The model never sees the explicit clause-to-guideline mapping, but the rule text is an abstraction of the very clauses being predicted, so the high void recall demonstrates in-sample rule matching rather than generalization to unseen contracts. The paper frames the result as showing 'the potential of LLMs to assist lawyers' without a held-out evaluation or guidelines drafted independently of the test clauses; the limitations section does not acknowledge this reverse-engineering. The full-text variant similarly uses the gold annotations to select the exact sources that make each void clause void, which is disclosed as a 'perfect retrieval' assumption and is therefore a transparent oracle condition rather than a concealed circularity, but it still prevents the full-text results from being an independent prediction. No external benchmark or independent guideline validation is provided. These two design choices together mean the central positive claim rests on test-set-derived inputs, warranting a partial-circularity score of 6. The dataset, code, and multi-model comparison remain useful contributions, but the 80% F1 should not be read as evidence of generalizable legal subsumption.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's central result depends on two hand-made knowledge resources: the examination guidelines and the source-selection annotations. Both are derived from the same dataset used for evaluation, which is the main circularity burden. No new entities are introduced, and the only assumptions are domain-specific choices about what constitutes a fair measurement of subsumption.

free parameters (1)
  • Examination guidelines (24 rules) = 24 hand-written rules derived from the void clauses in the dataset
    The guidelines are the main context that yields the 80% F1 result. They were created by grouping the dataset's own void clauses (Section 4.1), so they are effectively fitted to the evaluation set. This is the paper's central data-dependent construct.
assumptions (3)
  • domain assumption Legal subsumption can be meaningfully quantified by classification metrics without analyzing the LLMs' explanations.
    Stated in Section 5: 'our methodology assumes that the subsumption ability can be meaningfully quantified without it.' The paper publishes explanations but does not analyze them.
  • domain assumption The filtered dataset of 891 samples and the definitions of valid/unfair/void are accepted as ground truth.
    The paper relies on the annotations from the prior Wardas et al. dataset (Section 4) without independent verification.
  • domain assumption Providing the exact legal sources for each void clause (perfect retrieval) isolates subsumption ability.
    Section 5.3: the full-text variant uses ground-truth annotations to select sources, assuming a perfect retrieval system. This is not a realistic deployment setting but is treated as an experimental control.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs for Legal Subsumption in German Employment Contracts." pith.science (2026). https://pith.science/paper/YMRHPDO4

@misc{pith2026250701734,
  author       = {Pith},
  title        = {Pith review of: LLMs for Legal Subsumption in German Employment Contracts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMRHPDO4}},
  note         = {Machine review of arXiv:2507.01734}
}
read the original abstract

Legal work, characterized by its text-heavy and resource-intensive nature, presents unique challenges and opportunities for NLP research. While data-driven approaches have advanced the field, their lack of interpretability and trustworthiness limits their applicability in dynamic legal environments. To address these issues, we collaborated with legal experts to extend an existing dataset and explored the use of Large Language Models (LLMs) and in-context learning to evaluate the legality of clauses in German employment contracts. Our work evaluates the ability of different LLMs to classify clauses as "valid," "unfair," or "void" under three legal context variants: no legal context, full-text sources of laws and court rulings, and distilled versions of these (referred to as examination guidelines). Results show that full-text sources moderately improve performance, while examination guidelines significantly enhance recall for void clauses and weighted F1-Score, reaching 80\%. Despite these advancements, LLMs' performance when using full-text sources remains substantially below that of human lawyers. We contribute an extended dataset, including examination guidelines, referenced legal sources, and corresponding annotations, alongside our code and all log files. Our findings highlight the potential of LLMs to assist lawyers in contract legality review while also underscoring the limitations of the methods presented.

Figures

Figures reproduced from arXiv: 2507.01734 by the authors.

Figure 1
Figure 1. presents the confusion matrix for GPT-4o when using ex￾amination guidelines. Void clauses that are misclassified are more frequently placed in the unfair class rather than the valid class. Sim￾ilarly, misclassified valid clauses tend to be assigned to the unfair class rather than the void class. This indicates that the model rarely makes entirely incorrect predictions (e.g. predicting valid as void or vice versa). I… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 27 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Daniel Braun. 2021. Automated Semantic Analysis, Legal Assessment, and Summa- rization of Standard Form Contracts . Ph. D. Dissertation. Technische Universität München

    work page 2021

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  4. [4]

    Bruce G Buchanan and Thomas E Headrick. 1970. Some speculation about artificial intelligence and legal reasoning. Stan. L. Rev. 23 (1970), 40

    work page 1970

  5. [5]

    James E Cabral, Abhijeet Chavan, Thomas M Clarke, and John Greacen. 2012. Using technology to enhance access to justice. Harv. JL & Tech. 26 (2012), 241

    work page 2012

  6. [6]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 [cs.AI]

    arXiv 2024

  7. [7]

    Corinna Coupette, Janis Beckedorf, Dirk Hartung, Michael Bommarito, and Daniel Martin Katz. 2021. Measuring law over time: A network analytical frame- work with an application to statutes and regulations in the United States and Germany. Frontiers in Physics 9 (2021), 658463

    work page 2021

  8. [8]

    DeepSeek, Inc. 2025. DeepSeek-R1 Release. https://api-docs.deepseek.com/news/ news250120 Accessed: 2025-01-26

    work page 2025

Show all 39 references
  1. [9]

    Nick Goodson and Rongfei Lu. 2023. Intention and Context Elicitation with Large Language Models in the Legal Aid Intake Process.arXiv preprint arXiv:2311.13281 (2023)

    2023 arXiv

  2. [10]

    Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268 (2021)

    2021 arXiv

  3. [11]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv preprint arXiv:2404.06654 (2024)

    2024 arXiv

  4. [12]

    Nathaniel Huber-Fliflet, Jianping Zhang, Peter Gronvall, Fusheng Wei, and Philip Spinelli. 2024. Experimental Study of In-Context Learning for Text Classification and Its Application to Legal Document Review in Construction Delay Disputes. In 2024 IEEE International Conference...

    2024

  5. [13]

    Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo

  6. [14]

    Daniel Martin Katz, Corinna Coupette, Janis Beckedorf, and Dirk Hartung. 2020. Complex societies and the growth of the law.Scientific reports 10, 1 (2020), 18737

    2020

  7. [15]

    Daniel Martin Katz, Dirk Hartung, Lauritz Gerlach, Abhik Jana, and Michael J Bommarito II. 2023. Natural language processing in the legal domain. arXiv preprint arXiv:2302.12039 (2023)

    2023 arXiv

  8. [16]

    Jeehee Lee, June-Seong Yi, and JeongWook Son. 2019. Development of automatic- extraction model of poisonous clauses in international construction contracts using rule-based NLP. Journal of Computing in Civil Engineering 33, 3 (2019), 04019003

    2019

  9. [17]

    Spyretta Leivaditi, Julien Rossi, and Evangelos Kanoulas. 2020. A benchmark for lease contract review. arXiv preprint arXiv:2010.10386 (2020)

    2020 arXiv

  10. [18]

    Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans- Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. 2019. CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law 27 (2019), 117–139

    2019

  11. [19]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  12. [20]

    Robert Mahari, Dominik Stammbach, Elliott Ash, and Alex’Sandy’ Pentland

  13. [21]

    Shubham Kumar Nigam and Aniket Deroy. 2023. Fact-based Court Judgment Prediction. In Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation. 78–82

    2023

  14. [22]

    Shubham Kumar Nigam, Aniket Deroy, Subhankar Maity, and Arnab Bhat- tacharya. 2024. Rethinking legal judgement prediction in a realistic scenario in the era of large language models. arXiv preprint arXiv:2410.10542 (2024)

    2024 arXiv

  15. [23]

    OpenAI. 2024. Introducing OpenAI o1. https://openai.com/o1/ Accessed: 2025- 01-26

    2024

  16. [24]

    Ali Hakimi Parizi, Yuyang Liu, Prudhvi Nokku, Sina Gholamian, and David Emerson. 2023. A Comparative Study of Prompting Strategies for Legal Text Classification. In Proceedings of the Natural Legal Language Processing Workshop

    2023

  17. [25]

    Ralf Poscher. 2011. Ambiguity and vagueness in legal interpretation. (2011)

    2011

  18. [26]

    Jonathan Roberts, Kai Han, and Samuel Albanie. 2024. Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks? arXiv preprint arXiv:2411.05000 (2024)

    2024 arXiv

  19. [27]

    TYS Santosh, Kevin D Ashley, Katie Atkinson, and Matthias Grabmair. 2024. Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need? arXiv preprint arXiv:2406.10974 (2024)

    2024 arXiv

  20. [28]

    TYSS Santosh, Mohamed Hesham Elganayni, Stanisław Sójka, and Matthias Grabmair. 2024. Incorporating precedents for legal judgement prediction on european court of human rights cases. arXiv preprint arXiv:2409.18644 (2024)

    2024 arXiv

  21. [29]

    TYS Santosh, Shanshan Xu, Oana Ichim, and Matthias Grabmair. 2022. Decon- founding legal judgment prediction for European court of human rights cases towards better alignment with experts. arXiv preprint arXiv:2210.13836 (2022)

    2022 arXiv

  22. [30]

    Dan Schumacher and Anthony Rios. 2024. Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4. arXiv preprint arXiv:2404.01961 (2024)

    2024 arXiv

  23. [31]

    Dong Shu, Haoran Zhao, Xukun Liu, David Demeter, Mengnan Du, and Yongfeng Zhang. 2024. LawLLM: Law large language model for the US legal system. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 4882–4889

    2024

  24. [32]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  25. [33]

    Sebastian Urbina. 2002. Legal method and the rule of law. Vol. 59. Springer Science & Business Media

    2002

  26. [34]

    Josef Valvoda and Ryan Cotterell. 2024. Towards Explainability in Legal Outcome Prediction Models. arXiv preprint arXiv:2403.16852 (2024)

    2024 arXiv

  27. [35]

    Oliver Wardas and Florian Matthes. 2025. AI-assisted German Employ- ment Contract Review: A Benchmark Dataset. preprint TUM (2025). ICAIL, June 16–20, 2025, Chicago, IL Wardas et al. https://wwwmatthes.in.tum.de/file/1au0alwr7qbf1/Sebis-Public-Website/- /AI-assisted-German-Emp...

    2025

  28. [36]

    xAI. 2024. Grok-2 Beta Release. https://x.ai/blog/grok-2 Accessed: 2025-01-26

    2024

  29. [37]

    ###con- tent###

    Fangyi Yu, Lee Quartey, and Frank Schilder. 2023. Exploring the effectiveness of prompt engineering for legal reasoning tasks. In Findings of the Association for Computational Linguistics: ACL 2023. 13582–13596. A Appendix A.1 Prompt Templates All prompts are written in German...

    2023

  30. [2023]

    arXiv preprint arXiv:2310.14346 (2023)

    The law and NLP: Bridging disciplinary disconnects. arXiv preprint arXiv:2310.14346 (2023)

    2023 arXiv

  31. [2024]

    Philosophical Transactions of the Royal Society A 382, 2270 (2024), 20230254

    Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A 382, 2270 (2024), 20230254

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.