{"id":"78815da0-1600-4b75-9b41-a68f1299e8d3","arxiv_id":"2412.15902","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Pre-trained open LLMs underperform bag-of-words baselines on German Gutachtenstil identification and legal essay grading, but retrieval-based example selection narrows the gap on simpler tasks.","lead":"Open-source LLMs can do simple German legal text analysis, but they fail to beat a bag-of-words classifier at recognizing Gutachtenstil components or grading full legal essays, no matter how they are prompted. The paper also introduces a retrieval-based method that picks better few-shot examples and a new dataset of graded German criminal-law essays, which is useful for anyone building or buying legal-education AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SPWSLE negative result rests on an estimated label mapping (§4.2); if e5/e6/e7 are misassigned, the class-level F1 and the headline baseline gap in Tables 2–3 must be recomputed.","rationale":"I read the full text. The paper's most important empirical contribution is the negative result that LLMs cannot match BoW baselines on complex German legal analysis. Of the four datasets, SPWSLE is the one used for the 'Gutachtenstil' headline and for the RAG and CoT ablations. The label mapping is an explicit estimate; the authors flag it in §4.2. Because the mapping determines what the classes mean, every aggregate and per-class number for this task depends on it. This is a correctness risk, not merely a citation or clarity issue: if e3 is non-empty, or if e5/e6/e7 are swapped, then the LMs are being scored against a different ground truth than the dataset authors intended. The code release does not by itself resolve this, because the mapping is an annotation-level fact, not a code-level fact. I agree with the reader's weakest_assumption. Other issues exist—the abstract's low-data improvement claim is not directly tabulated, and GSHA results are only shown in a figure—but the mapping is the most load-bearing because it sits under the primary negative finding. The recommended check would settle it; until then, a conditional verdict is appropriate. I do not see an internal inconsistency that would warrant rejection, and the paper's own pseudonymization and RAG-brittleness tests provide useful supporting evidence.","tokens_in":11382,"tokens_out":5137,"duration_ms":44465,"concrete_test":"Contact the SPWSLE authors (Weber et al., 2023) or inspect the released dataset/code to obtain the official e1-e7 to Gutachtenstil mapping. Then re-run the joint-task evaluation of §4.2 with the verified mapping for BoW+SVM and Llama 3 10RAG. Report macro F1 and per-class F1 for all seven labels. If the Llama 3 10RAG macro-F1 deficit versus BoW+SVM changes by under 5 relative points, the central result stands; if it changes by more than 10 points, or if any class flips from near-zero to substantial F1, the paper's negative claim must be revised to the corrected task.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative claim on the SPWSLE Gutachtenstil task is computed on a label-space reconstruction. Section 4.2 says of the original annotation scheme: 'It is not exactly clear how they do this' and 'no mapping ... is included. Our estimate is the following: e1=Major Claim, e2=Conclusion, e4=Definition, e5=Subsumption, e6=Legal Claim, e7=Premise, with e3 remaining unused throughout all data.' Every result in Tables 2, 3, 9-12 inherits this estimate. If e3 is used at all, or if e5/e6/e7 are permuted relative to the true scheme, then the near-zero Legal Claim F1 values and the headline gap (BoW+SVM macro F1 .761 vs. Llama 3 10RAG .580) measure a different task from the one Weber et al. (2023) annotated. The negative conclusion is thus not independently checkable from the text; a corrected mapping could preserve, shrink, or even reverse the gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the suitability of pre-trained generative LLMs (Llama 3 70B Instruct, Mixtral 8x7B Instruct, GPT-3.5 Turbo) for German legal analysis in educational contexts. It uses four datasets: two German argument-mining corpora (SPWSLE, CIMT) and two essay-scoring collections (a newly introduced Graded Strafrecht Hausarbeiten, GSHA, and ASAP set 8). The main results are that LLMs underperform a BoW+SVM baseline on SPWSLE Gutachtenstil classification (Table 3: BoW macro F1 0.761 vs. best LLM 0.580), perform near random on GSHA, but are competitive or better on CIMT and ASAP. The paper also introduces a RAG-based prompt example selection method that improves few-shot performance, evaluates Chain-of-Thought prompting, and reports a pseudonymization control experiment.","tokens_in":11657,"tokens_out":5597,"duration_ms":46160,"significance":"The paper addresses a timely and practically important question: can open-weight generative LLMs handle structured German legal analysis in education? If the negative results are robust, they provide a useful counterpoint to uncritical claims of LLM competence in specialized domains. The introduction of the GSHA dataset is a concrete contribution, the RAG-based example selection is systematically studied, and the pseudonymization experiment (Table 5) is a clever control for internalized category knowledge. However, the headline SPWSLE result rests on an explicitly estimated label mapping and a single data split, and the abstract's low-data superiority claim is not supported by reported baselines. The significance is therefore conditional on these methodological points being resolved.","major_comments":[{"comment":"The SPWSLE comparison relies on an estimated mapping from the original annotation scheme to the seven Gutachtenstil categories. The text states, \"It is not exactly clear how they do this... Our estimate is the following: e1=Major Claim, e2=Conclusion, e4=Definition, e5=Subsumption, e6=Legal Claim, e7=Premise, with e3 remaining unused throughout all data.\" Every SPWSLE result, including the headline gap (BoW+SVM macro F1 0.761 vs. Llama 3 10RAG 0.580 in Table 3), is computed on this reconstruction. If e3 is used or e5/e6/e7 are permuted relative to the true scheme, the class-level F1 values and the overall gap would measure a different task. Please verify the mapping by contacting the original authors or inspecting annotation guidelines, or report a sensitivity analysis over plausible alternative mappings. At minimum, the caveat should be stated prominently wherever \"cannot match\" is claimed.","section":"§4.2 (Evaluation Methodology) and Tables 2, 3, 9–12"},{"comment":"The abstract claims, \"Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case.\" No BoW+SVM baseline is reported for the low-data scenarios. Table 4 compares only Llama 3 variants (CoT vs. result-only) at 80%, 0.3%, and 0% training data, and Tables 6–7 always use the full training set for the BoW baseline. The \"throughout\" claim is therefore unsupported. Please add low-data BoW baselines (e.g., training the SVM on the same subsamples) or remove/qualify the claim to the specific tasks where it actually holds.","section":"Abstract and §4.3, Table 4"},{"comment":"All SPWSLE results are computed on a single 20% test split, with no variance estimates or cross-validation. Given the dataset size (382 documents) and the strong negative conclusion (\"cannot match\" in §4.3), the results could depend on the idiosyncrasies of one split. Please report k-fold cross-validation or repeated random splits with standard deviations for the main comparison, at least for the BoW+SVM baseline and the best LLM configuration.","section":"§4.3, Tables 2–3 and 9"},{"comment":"The claim that LLMs \"cannot achieve prediction quality beyond random guessing\" on GSHA is not supported by reproducible numbers. Figure 6 is referenced but no quantitative correlation/accuracy values or a formal random baseline are given in the text. Please report the actual Spearman/Pearson correlations and accuracies with standard errors, and define what \"random guessing\" means for this 0–18 grading scale (e.g., majority-class or uniform random prediction).","section":"§4.5, Figure 6"}],"minor_comments":[{"comment":"The phrase \"not no par\" appears to be a typo for \"not on par.\"","section":"§4.3.2"},{"comment":"The word \"Detauled\" should be \"Detailed.\"","section":"Table 11 caption"},{"comment":"Both rows in Table 10 are labeled \"RAG\" in the table body, but the caption indicates that the second row is \"Inverse RAG.\" Please correct the row labels for clarity.","section":"Table 10"},{"comment":"Since GSHA is a newly introduced dataset containing student work, please specify the anonymization/pseudonymization procedure and the intended data-release license, in addition to the consent statement already present.","section":"§4.1.2 and Appendix"},{"comment":"The LiveBench scores in Table 8 are presented but never explicitly used in the analysis. Either connect them to the interpretation of the German-task results or remove them to tighten the exposition.","section":"§3 Methods, Table 8"},{"comment":"The Generated Artificial Rationales (GAR) method is introduced as a contribution, but Table 4 shows it never improves over the result-only prompt and in some cases degrades performance. Consider framing GAR explicitly as a negative result rather than part of the proposed method.","section":"§3.5 and Table 4"},{"comment":"Figure 6 is mentioned in §4.5 but does not appear in the text extract; ensure the final version includes the figure and that its axes and error bars are legible.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core, but the two main weaknesses — the estimated label mapping for SPWSLE and the unsupported low-data claim in the abstract — are load-bearing for the central message. If the authors can verify the mapping (or show sensitivity to it) and add multi-split results, the paper would be a useful contribution to the emerging evaluation literature on LLMs in legal education. The GSHA dataset is a positive addition, though it currently lacks reliability information. I would advise the editor to request these revisions before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of Wendlinger et al. The paper's core finding — that open foundational LLMs with few-shot prompting cannot beat a bag-of-words SVM on Gutachtenstil component classification and are near random on full legal essay grading — is plausible and worth reporting. The new GSHA dataset is a genuine contribution, and the RAG-based example selection (GAR) plus the pseudonymization control are thoughtful additions. I'd treat the negative result as a useful measurement for anyone building legal edtech on open models.\n\nThe biggest soft spot is the SPWSLE comparison. The authors reconstructed the label mapping from Weber et al.'s annotation scheme, and they say so: 'It is not exactly clear how they do this... Our estimate is the following.' That means every class-level F1 in Tables 2, 3, 9–12 is computed on an estimated task. The overall conclusion may survive a corrected mapping, but the headline gap (BoW+SVM .761 vs. Llama 3 10RAG .580) is not independently checkable from the paper. The paper needs to publish the mapping, check its consistency, or run a sensitivity analysis.\n\nSecond, the abstract claims LLMs 'improve upon the baseline in scenarios with little or no labeled data,' but no low-data Bag-of-Words baseline appears anywhere in the reported tables. Table 4 compares CoT and result-prompt variants across data availability, but the BoW baseline is only trained on the 80% split. That claim is currently unsupported.\n\nThird, GSHA results are reported as a figure only, with no numbers, and the dataset is not linked. For a new dataset presented as a transfer learning challenge, that is thin. The SPWSLE experiments use a single 20% split, so no variance estimate there either.\n\nThese are all fixable. The RAG brittleness test and the pseudonymization check are well done, and the CIMT and ASAP results give useful context. The paper is a solid applied negative result for a niche domain, not a breakthrough. It deserves a serious referee who knows Weber et al. and German legal annotation, but it needs revision before acceptance. I'd send it to review rather than desk reject; the new data and the clear negative finding justify referee time.","headline":"A useful negative result for German legal edtech — open LLMs fall below BoW on Gutachtenstil classification — but the SPWSLE label mapping is reconstructed and the low-data claim is unsupported.","tokens_in":12180,"tokens_out":2589,"would_cite":false,"duration_ms":21968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source LLMs are competent at simple German legal argument mining, but fail to match a bag-of-words classifier on Gutachtenstil recognition and cannot grade full German legal essays beyond random guessing.","keywords":["German legal education","Gutachtenstil","large language models","argument mining","automated essay scoring","retrieval augmented generation","chain-of-thought prompting","German NLP"],"falsifier":"Run the SPWSLE classification using the original annotation mapping (or a mapping confirmed by the dataset creators) instead of the paper's estimated mapping; if the best LLM with RAG-selected shots then reaches or passes the BoW+SVM macro F1 of 0.761, the claim that LLMs cannot match extractive baselines on this task would be overturned. For the essay-grading claim, a concrete test would be to collect a larger corpus of graded German legal essays with multiple annotators and check whether any open pre-trained LLM produces a Spearman correlation with grades significantly above zero.","tokens_in":11215,"feed_emoji":"⚖️","tokens_out":8992,"duration_ms":69191,"temperature":0.7,"pith_summary":"This paper asks whether open-source pre-trained LLMs can perform legal analysis in German legal education, and answers with a qualified no. It shows the models handle simpler tasks—classifying public-feedback argument components and scoring English student essays—about as well as or better than a bag-of-words baseline. But for the task that defines legal education in Germany, recognizing the components of Gutachtenstil in student legal case solutions, all tested LLMs fall well behind a linear bag-of-words classifier even when given retrieved examples and chain-of-thought prompts, and for grading complete German legal exam essays they predict grades no better than random. The paper introduces a retrieval-augmented generation based method for selecting few-shot examples that improves LLM predictions in data-rich settings, and argues that careful example selection matters more than reasoning prompts. The value is a clear empirical boundary for current open LLMs in a high-stakes, low-resource educational domain.","feed_headline":"Open LLMs can't beat a bag-of-words baseline on German legal style","feed_subtitle":"Even with retrieved examples and chain-of-thought, the best LLM hit 0.580 macro F1 versus 0.761 for the simple baseline.","key_machinery":"The central object is the joint Gutachtenstil classification task on the SPWSLE dataset, in which each sentence of a German law student's case solution must be labeled as Major Claim, Conclusion, Definition, Subsumption, Legal Claim, Premise, or None. 'Gutachtenstil' is the German legal appraisal style—major claim, definition, subsumption, conclusion—that law students are taught to structure their reasoning. The argument is carried by the controlled comparison of this task against a linear SVM with Bag-of-Words features as the extractive baseline, and by the prompting machinery tested around it: zero-shot and few-shot prompts, Chain-of-Thought prompts (with pattern matching to recover the final category), Generated Artificial Rationales (auto-generated reasoning examples adapted from Auto-CoT), pseudonymization of category names to test internalized knowledge, and RAG-based selection of example shots using dense embeddings (jina-embeddings-v2-base-de). The RAG selection mechanism is the one intervention that consistently improves LLM predictions in high-data settings, and the pseudonymization ablation shows the models rely on genuine category knowledge rather than surface patterns.","core_discovery":"The paper's central claim is that open-source pre-trained LLMs have enough instruction capability and German legal background knowledge for some legal-analysis tasks, but not for the tasks that legal education depends on. On the joint Gutachtenstil component classification of SPWSLE student case solutions, the best tested LLM (Llama 3 with ten RAG-selected example shots) reaches a macro F1 of 0.580 and accuracy of 0.710, while a Bag-of-Words linear SVM baseline reaches 0.761 and 0.784; the LLMs fall particularly short on the subsumption-related categories (Premise, Legal Claim, Subsumption). On the new Graded Strafrecht Hausarbeiten dataset, where full student legal opinions must be graded on a 0-18 scale, the models produce predictions no better than random guessing, even with extended context, simplified grading scales, and chain-of-thought prompts. The authors interpret this as a limitation of instruction following and German-language proficiency in current open models, and they show that retrieval-augmented selection of prompt examples is the most effective mitigation, while Chain-of-Thought helps only in the zero-shot case.","pith_inferences":["If the original SPWSLE annotation mapping differs from the authors' estimate, the headline negative result would need recomputation; a direct follow-up is to obtain the documented mapping from the dataset authors and rerun both baseline and LLM conditions.","The authors note in the Limitations section that case-to-annotation linking is not present in the documents, so knowledge transfer to new within-domain tasks was untested; a concrete extension is to annotate a second set of legal cases and measure whether RAG-selected shots from one case still help on another.","The random-guessing result on GSHA could partly reflect annotation noise and the small 76-document corpus rather than model incapacity; a testable extension is to obtain multiple independent grades for the same essays and compare the model's rank correlation with the human inter-annotator correlation.","A practical hybrid suggested by the paper's numbers is to keep the BoW+SVM classifier for Gutachtenstil component extraction and reserve RAG-prompted LLMs for zero-shot or low-label settings, rather than replacing the baseline outright."],"forward_implications":["For Gutachtenstil component classification, practitioners should expect open pre-trained LLMs to underperform a cheap Bag-of-Words+SVM baseline even with retrieved examples, so the extractive baseline should not be discarded.","On the simpler tasks (CIMT public-feedback argument mining and English ASAP essay scoring), open LLMs with a few example shots match or beat the BoW baseline, and Chain-of-Thought prompting helps in the zero-shot case, making them usable in low-label regimes.","RAG-based example selection is a strong lever: choosing similar examples changes macro F1 by 14.4% on SPWSLE, and inverse retrieval drops performance below random shot selection, so prompt-example quality should be treated as a first-class design variable.","Full legal essay grading on the new GSHA dataset is beyond current open pre-trained LLMs; achieving it will require either substantially more capable models, fine-tuning on much larger graded corpora, or a different task formulation.","Even in the easier two-tiered extraction setting, LLMs trail the BERT-based and BoW classifiers, indicating the difficulty is inherent to component recognition rather than the joint evaluation scheme."],"supporting_citations":[{"why":"Provides the SPWSLE dataset of German legal case solutions and the BERT-based extractive classifier whose results are compared as a reference point.","marker":"(Weber et al., 2023)"},{"why":"Provides the CIMT public-feedback argument-mining dataset and the SVM result that the paper tries to reproduce.","marker":"(Romberg and Conrad, 2021)"},{"why":"Provides the ASAP Automated Essay Scoring dataset used for the English essay-scoring comparison.","marker":"(Hamner et al., 2012)"},{"why":"Supplies the Retrieval Augmented Generation method that the paper adapts to select few-shot prompt examples by dense retrieval.","marker":"(Lewis et al., 2020)"},{"why":"Supplies Chain-of-Thought prompting, whose zero-shot benefit and few-shot limitations the paper measures.","marker":"(Wei et al., 2022)"},{"why":"Establishes few-shot prompting as the basis for the example-shot prompts used throughout the experiments.","marker":"(Brown et al., 2020)"},{"why":"Supplies Auto-CoT's rationale-generation and selection criterion, which the paper adapts into Generated Artificial Rationales.","marker":"(Zhang et al., 2022)"},{"why":"Shows that similar rather than random example shots improve legal argument-mining performance, motivating the RAG selection strategy.","marker":"(Al Zubaer et al., 2023)"},{"why":"Provides LiveBench scores used to choose Llama 3, GPT-3.5, and Mixtral for their complementary reasoning and language strengths.","marker":"(White et al., 2024)"}],"fun_headline_variants":["Bag-of-words beats open LLMs on German legal style","LLMs can't match bag-of-words on legal style analysis","Even RAG examples leave LLMs behind bag-of-words","German legal style: simple baseline outperforms LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline negative result about the German appraisal-style task rests on the authors' guess about how the original dataset labels map to the seven Gutachtenstil categories; the original paper does not document this mapping, so a different mapping would change the comparison.","fun_headline_variants_meta":{"raw":{"variants":["Bag-of-words beats open LLMs on German legal style","LLMs can't match bag-of-words on legal style analysis","Even RAG examples leave LLMs behind bag-of-words","German legal style: simple baseline outperforms LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001076,"raw_usage":{"total_tokens":4504,"prompt_tokens":944,"completion_tokens":3560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":3490}},"tokens_in":560,"tokens_out":3560,"duration_ms":24802,"temperature":1.0,"reasoning_tokens":3490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:59:08.934664+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the SPWSLE classification using the original annotation mapping (or a mapping confirmed by the dataset creators) instead of the paper's estimated mapping; if the best LLM with RAG-selected shots then reaches or passes the BoW+SVM macro F1 of 0.761, the claim that LLMs cannot match extractive baselines on this task would be overturned. For the essay-grading claim, a concrete test would be to collect a larger corpus of graded German legal essays with multiple annotators and check whether any open pre-trained LLM produces a Spearman correlation with grades significantly above zero.","supporting_citations":[],"review_version":1}