{"id":"6d20c0f3-066c-4775-950e-277d1735a4a1","arxiv_id":"2507.01734","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs reach 80% weighted F1 on German employment contract clause review when given lawyer-distilled examination guidelines, but lag human lawyers when reading full legal sources.","lead":"Researchers tested five large language models on classifying clauses in German employment contracts as valid, unfair, or void, under three kinds of legal context. The finding that lawyer-written summaries of legal rules work far better than full legal texts could guide how AI legal assistants are built.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Examination guidelines were derived from the same void clauses used for evaluation (Sec. 4.1), so the reported 80% F1 and void recall may measure matching against a test-set-derived rule base, not generalizable legal subsumption; a held-out evaluation is needed.","rationale":"I read the paper in good faith. The experimental design is unusually transparent: the dataset extension, prompts, and log files are released, and the paper clearly describes how the guidelines were created. The full-text condition is also honest about the perfect-retrieval assumption. The most load-bearing concern is not an internal inconsistency but an external validity threat. The guideline condition produces the paper's headline positive claim, and its evaluation is contaminated by the fact that the guidelines were derived from the same clauses on which they are tested. This does not require the authors to have hidden anything; it is a design property visible from Section 4.1. Without a held-out evaluation, the 80% F1 and void recall cannot be taken as evidence that a deployed assistant would perform at that level on new contracts. The full-text lag claim is interesting but secondary; if anything, the poor full-text performance despite gold-selected sources supports the paper's own limitation statement. The reader's weakest assumption identifies exactly this issue, and I agree. I do not see a reason to change the verdict; CONDITIONAL remains appropriate, contingent on the held-out validation.","tokens_in":13060,"tokens_out":5468,"duration_ms":59588,"concrete_test":"Hold out a random 25% of the 112 void clauses (with matched valid/unfair samples) before the guideline-construction step; have the two lawyers create guidelines using only the remaining 75% of clauses, then evaluate GPT-4o on the held-out 25% with those guidelines, repeating across several splits. If average void recall and weighted F1 on held-out clauses fall substantially below Table 3 (e.g., recall below 0.65), the reported 0.80 recall is inflated by test-set-derived guidelines. As a complementary check, apply the published guidelines to an external set of German employment contract clauses annotated by lawyers; a large drop in void recall would likewise confirm the generalization gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs with lawyer-written examination guidelines achieve 80% weighted F1 and high void recall (Sec. 5.4, Table 3) rests on an evaluation where the guidelines were constructed from the dataset's own void clauses. Section 4.1 states that the two lawyers 'grouped clauses together which were void for the same reason' and formulated one examination guideline per group. The guidelines are therefore an abstraction of the 112 void clauses that constitute the positive class in the evaluation set. Although the model is not shown the clause-to-guideline mapping, the guideline texts themselves were fit to those clauses. Under this design, a model can achieve high void recall by matching clause wording to the rule text derived from that same wording, rather than by performing legal subsumption from an independent legal knowledge base. The claim of 'potential to assist lawyers' requires that guidelines generalize to unseen clauses, as they would in practice where they are drafted from laws and court rulings rather than from the contracts being reviewed. The paper acknowledges that guidelines simplify legal context (Sec. 6.3) but does not address this reverse-engineering issue, making the headline result optimistic as evidence of generalizable subsumption ability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends an existing German employment contract clause dataset with lawyer-created examination guidelines, references to laws and court rulings, and full-text legal sources, then evaluates five LLMs on a three-way legality classification task (valid/unfair/void) under three context variants: no legal context, full-text legal sources, and distilled examination guidelines. The main empirical claim is that providing examination guidelines yields a weighted F1 of about 80% and high recall for void clauses, while full-text sources improve over no context but remain below the performance attributed to human lawyers. The authors release the extended dataset, code, and all log files. The paper positions the guideline condition as evidence that LLMs can assist lawyers in contract legality review, while the full-text condition is used to argue that LLMs still lag behind humans in direct legal subsumption from original sources.","tokens_in":13253,"tokens_out":4805,"duration_ms":59673,"significance":"If the reported results were robust and generalizable, the paper would make a useful contribution by demonstrating a concrete human-in-the-loop setup for German employment contract review and by releasing a valuable resource: 24 examination guidelines, 41 referenced legal sources, and annotations linking void clauses to guidelines. The release of code and log files is a genuine strength and supports reproducibility of the experiments as run. However, the significance of the central claim is materially limited by the evaluation design: the examination guidelines were constructed from the same void clauses that constitute the test set, so the guideline condition may measure rule-matching against a test-set-derived rule base rather than generalizable legal subsumption. The comparison with human lawyers is also asserted without a reported human-subject experiment. These issues mean the paper currently establishes an interesting experimental comparison but not the broader capability claim stated in the abstract.","major_comments":[{"comment":"The 24 examination guidelines were created by having lawyers group the dataset's own 112 void clauses and formulate one rule per group (Section 4.1). Evaluating those same clauses with those guidelines in the prompt therefore measures the model's ability to match clause text to a rule abstracted from that same text, not subsumption under an independently formulated body of legal knowledge. The reported 80% weighted F1 and void recall values up to 98% (Table 3) are consistent with reverse-engineering the test set and do not establish generalizable subsumption ability. I recommend a held-out evaluation in which guidelines are derived only from training clauses and applied to unseen clauses, or at minimum a per-guideline analysis reporting how many void clauses are near-paraphrases of the guideline text assigned to them.","section":"§4.1, §5.4, Table 3"},{"comment":"The full-text variant uses gold annotations to supply exactly the references under which each void clause is void, while for non-void clauses it randomly selects two topic-matched sources. This is an oracle retrieval setup, which is acceptable for isolating subsumption, but it is not a neutral comparison against the guideline condition: both informative conditions derive their content from the same lawyer-constructed mapping from test clauses to legal rules. In addition, the instruction to classify a clause as void only if it can be subsumed under the given sources, combined with arbitrary source selection for valid and unfair clauses, may push predictions toward the majority 'valid' class. Please report full per-class confusion matrices for every condition and discuss how the oracle source selection affects the comparison between the guideline and full-text variants.","section":"§5.3, §5.4"},{"comment":"The paper repeatedly states that LLM performance with full-text legal sources is 'substantially below that of human lawyers,' but no human-subject experiment on this dataset or under the same evaluation protocol is reported. This claim is load-bearing for the conclusion that LLMs cannot yet perform unassisted legality review from original sources. The authors should add a human baseline measured on the same clauses and context variants, or reframe the claim as a comparison with expert-derived guidelines rather than with human performance.","section":"§5.4, §6.1"}],"minor_comments":[{"comment":"There are typos in the prose: 'lack behind' should be 'lag behind' in General Observations, and 'hightest' should be 'highest' in the GPT-4o Performance paragraph.","section":"§5.4"},{"comment":"No confidence intervals or repeated-run variance are reported even though API-based LLM outputs are stochastic and the seed parameter is not supported for DeepSeek-V3 and DeepSeek-R1. A brief statement about single-run results and their reproducibility would strengthen the empirical claims.","section":"Table 3, §5.2"},{"comment":"The 'hurt_rules' output field is only collected in the guideline condition, but the paper does not analyze whether the models' stated rule IDs match the gold guideline annotations. An analysis of these outputs would directly address whether high void recall is accompanied by correct identification of the applicable guideline.","section":"§5.2"},{"comment":"The missing DeepSeek-R1 cell in the full-text condition is explained, but Table 3 would benefit from a footnote summarizing how the uneven model-by-condition design affects cross-model comparisons.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The evaluation design is transparent and the released resources are valuable, but the central comparison is weakened by the test-set-derived guidelines and the unsupported human comparison. I saw no sign of misconduct; this is a methodological issue that can be addressed by a held-out evaluation or by substantially narrowing the claims. The paper is suitable for major revision rather than outright rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this paper is worth your time, but the headline number needs a caveat. The authors extended their earlier German employment-contract dataset with 24 lawyer-written examination guidelines, full-text legal sources, references, and per-clause annotations. That is real, reusable work, and they released code, logs, and full prompts. The evaluation is systematic: five LLMs, three context conditions, with a sensible design that isolates subsumption from retrieval by assuming perfect retrieval in the full-text condition.\n\nThe strongest result—80% weighted F1 and high void recall with examination guidelines—is undermined by a genuine circularity. Section 4.1 says the two lawyers grouped the dataset's void clauses by reason and formulated one guideline per group. The evaluation then classifies those same clauses. The guidelines are general legal rules, so this is not direct answer-key leakage, but the wording was informed by the test set. A model could achieve high recall by matching clause phrasing to rule text that was written from that same phrasing. The claim that LLMs can assist lawyers would be much better supported by a held-out evaluation using clauses the lawyers never saw when drafting the guidelines. As written, the 80% F1 is an optimistic estimate of generalizable subsumption ability.\n\nThe full-text condition also uses ground-truth annotations to select the exact sources that make each void clause void. The authors call this a perfect-retrieval assumption explicitly, so it is transparent, but it means those numbers are an upper bound, not a realistic deployment scenario. That is fine as an isolation strategy, but readers should not mistake it for end-to-end performance.\n\nTwo smaller issues. The abstract claims LLMs remain \"substantially below human lawyers,\" but no human baseline is measured in this study; that claim appears to come from prior work or intuition, not from this paper's data. And there are no error bars or significance tests across runs, so the differences between models (e.g., GPT-4o vs. DeepSeek-V3) could be noise.\n\nWho benefits? Anyone building legal NLP for German contracts will find the dataset and prompt templates useful, and the comparison of distilled guidelines vs. full-text sources is a data point worth having. The paper deserves a serious referee. I would send it to review with a request for either a held-out guideline evaluation or a revised interpretation that acknowledges the reverse-engineering risk. The dataset contribution alone justifies the review slot.","headline":"Solid, reproducible dataset work with a transparent evaluation, but the headline guideline-result is weakened by the guidelines being derived from the same clauses used for testing.","tokens_in":13791,"tokens_out":2315,"would_cite":true,"duration_ms":29397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLMs reach 80% weighted F1 and high void-clause recall on German employment contract clauses when given lawyer-written examination guidelines, while full-text legal sources still leave them below human lawyers.","keywords":["legal subsumption","German employment contracts","examination guidelines","large language models","in-context learning","contract legality classification","void clause detection","legal NLP"],"falsifier":"An independent evaluation on newly drafted employment clauses never seen by the guideline authors, with guidelines derived solely from statutes and court rulings, would settle whether the 80% weighted F1 reflects general subsumption ability; if void recall collapses to baseline on such held-out clauses, the guideline-based result is an artifact of rules reverse-engineered from the test set.","tokens_in":12831,"feed_emoji":"⚖️","tokens_out":5546,"duration_ms":60601,"temperature":0.7,"pith_summary":"Legal subsumption—deciding whether a particular contractual clause is void under the applicable law—is exactly the kind of text-heavy, high-stakes task that could benefit from language models. This paper tests whether large language models can perform that judgment on clauses from German employment contracts, comparing three settings: no legal context, the full text of laws and court rulings, and distilled rules written by lawyers (examination guidelines). The central result is that the lawyer-written guidelines outperform full-text sources, raising the best model's weighted F1 to 80% and recall for void clauses to 80% or higher for several models. With full-text sources, performance improves only moderately and remains substantially below that of human lawyers. The authors conclude that LLMs can usefully assist lawyers when legal knowledge is pre-digested, but cannot yet subsume directly under original legal sources.","feed_headline":"Distilled legal rules lift LLM contract review to 80% F1","feed_subtitle":"Feeding LLMs lawyers' distilled guidelines outperforms full legal texts, but full-text subsumption still trails human lawyers.","key_machinery":"The central object is the examination guideline: a short, distilled legal rule written by human lawyers that states the condition under which a group of clauses is void and lists the laws and court rulings from which the rule derives. In the guideline variant of the experiment, all 24 guidelines are placed in the system prompt, so the model must select applicable rules rather than retrieve them; in the full-text variant, original sources are supplied as if by perfect retrieval. The guidelines carry the argument because the gap between guideline-based and source-based performance is the paper's main evidence about where LLM legal subsumption currently fails.","core_discovery":"The paper's own claim is that the form in which legal knowledge is presented, not just model capability, determines whether LLMs can perform legal subsumption. Using 24 examination guidelines distilled from 41 laws and court rulings, the strongest model reaches an 80% weighted F1-score and 80% recall on void clauses, and several other models achieve void recall above 80% (up to 98% for the reasoning model DeepSeek-R1, albeit with lower precision). When the same models receive the original full-text sources, their performance falls, and the paper reads this gap as evidence that state-of-the-art LLMs can apply pre-digested legal rules but lag human lawyers at deriving subsumptions from primary legal materials. The paper also finds that every model improves when any legal context is supplied, which it takes as support for retrieval-augmented and knowledge-base approaches to legal NLP.","pith_inferences":["My inference: because the examination guidelines were distilled from the same clauses used in the test set, the high void recall may partly reflect rules tailored to those clauses; re-running with guidelines written only from statutes and rulings, or with newly drafted held-out clauses, would reveal how much general subsumption ability remains.","My inference: the guideline/full-text gap may be driven by input length and salience rather than legal reasoning alone; benchmarking with short extractive summaries of the same sources, or with guidelines of comparable length to source excerpts, would isolate the cause.","My inference: DeepSeek-R1's near-perfect void recall with low precision and a general bias toward void classifications suggests a natural human-in-the-loop design: use a high-recall model to flag candidate clauses and a higher-precision model to prepare the final draft."],"forward_implications":["If the guidelines-based result holds, a semi-automated contract review workflow that maintains a human-curated legal knowledge base and uses LLMs to produce initial drafts is within reach.","High void-clause recall—80% for the leading model, 82% for another, 98% for the reasoning model—attacks the most costly failure mode, because missing a void clause is the risk lawyers most need to avoid.","Because full-text sources only moderately improve predictions, applications built directly on raw laws and court rulings cannot yet be trusted for unassisted review, even under a perfect retrieval assumption.","The consistently poor performance on the unfair class means the models cannot yet separate unfair-but-valid clauses from void ones, so human oversight remains essential.","Providing any legal context improves every model's averaged F1 over internal knowledge alone, supporting retrieval-augmented and knowledge-base approaches to legal NLP."],"supporting_citations":[{"why":"Supplies the 891-clause German employment contract dataset with legality annotations that the paper extends with guidelines and sources.","marker":"[35]"},{"why":"Provides the GPT-4o model behind the best averaged F1 scores and the in-context learning approach used across experiments.","marker":"[1]"},{"why":"Used to rank and select the five evaluated models from a public leaderboard.","marker":"[6]"},{"why":"Reports GPT-4's strong bar exam performance, the human-lawyer comparison that the paper's weaker full-text results contrast with.","marker":"[13]"},{"why":"Motivates the paper's core premise by arguing that combining data-driven methods with explicit legal expertise is promising.","marker":"[27]"}],"fun_headline_variants":["Distilled legal rules boost LLM contract review to 80% F1","Condensed legal guidelines outperform full texts for LLM clause checks","LLMs ace distilled legal rules but fail full-text subsumption","For LLM contract review, distilled legal knowledge beats raw laws","Legal checklists lift LLM F1 to 80% on employment clauses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the examination guidelines, written by lawyers who grouped the dataset's own void clauses, serve as an independent source of legal knowledge rather than as a description of the test set.","fun_headline_variants_meta":{"raw":{"variants":["Distilled legal rules boost LLM contract review to 80% F1","Condensed legal guidelines outperform full texts for LLM clause checks","LLMs ace distilled legal rules but fail full-text subsumption","For LLM contract review, distilled legal knowledge beats raw laws","Legal checklists lift LLM F1 to 80% on employment clauses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1555,"prompt_tokens":948,"completion_tokens":607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":514}},"tokens_in":564,"tokens_out":607,"duration_ms":6292,"temperature":1.0,"reasoning_tokens":514,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:44:10.985171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent evaluation on newly drafted employment clauses never seen by the guideline authors, with guidelines derived solely from statutes and court rulings, would settle whether the 80% weighted F1 reflects general subsumption ability; if void recall collapses to baseline on such held-out clauses, the guideline-based result is an artifact of rules reverse-engineered from the test set.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 891-clause German employment contract dataset with legality annotations that the paper extends with guidelines and sources."},{"cited_title":"Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?","cited_arxiv_id":"2406.10974","evidence_quote":"Motivates the paper's core premise by arguing that combining data-driven methods with explicit legal expertise is promising."}],"review_version":1}