Pith. sign in

REVIEW 4 major objections 7 minor 16 references

On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Open-source LLMs are competent at simple German legal argument mining, but fail to match a bag-of-words classifier on Gutachtenstil recognition and cannot grade full German legal essays beyond random guessing.

desk verdict A useful negative result for German legal edtech — open LLMs fall below BoW on Gutachtenstil classification — but the SPWSLE label mapping is reconstructed and the low-data claim is unsupported. read the letter →

arxiv 2412.15902 v1 pith:AOHGHEWQ submitted 2024-12-20 cs.CL cs.AI

Lorenz Wendlinger , Christian Braun , Abdullah Al Zubaer , Simon Alexander Nonn , Sarah Gro{ss}kopf , Christofer Fellicious , Michael Granitzer This is my paper · ORCID
classification cs.CLcs.AI
keywords GermanlegaleducationGutachtenstillargelanguagemodelsargumentminingautomatedessayscoringretrievalaugmentedgenerationchain-of-thoughtpromptingNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether open-source pre-trained LLMs can perform legal analysis in German legal education, and answers with a qualified no. It shows the models handle simpler tasks—classifying public-feedback argument components and scoring English student essays—about as well as or better than a bag-of-words baseline. But for the task that defines legal education in Germany, recognizing the components of Gutachtenstil in student legal case solutions, all tested LLMs fall well behind a linear bag-of-words classifier even when given retrieved examples and chain-of-thought prompts, and for grading complete German legal exam essays they predict grades no better than random. The paper introduces a retrieval-augmented generation based method for selecting few-shot examples that improves LLM predictions in data-rich settings, and argues that careful example selection matters more than reasoning prompts. The value is a clear empirical boundary for current open LLMs in a high-stakes, low-resource educational domain.

What carries the argument

The central object is the joint Gutachtenstil classification task on the SPWSLE dataset, in which each sentence of a German law student's case solution must be labeled as Major Claim, Conclusion, Definition, Subsumption, Legal Claim, Premise, or None. 'Gutachtenstil' is the German legal appraisal style—major claim, definition, subsumption, conclusion—that law students are taught to structure their reasoning. The argument is carried by the controlled comparison of this task against a linear SVM with Bag-of-Words features as the extractive baseline, and by the prompting machinery tested around it: zero-shot and few-shot prompts, Chain-of-Thought prompts (with pattern matching to recover the final category), Generated Artificial Rationales (auto-generated reasoning examples adapted from Auto-CoT), pseudonymization of category names to test internalized knowledge, and RAG-based selection of example shots using dense embeddings (jina-embeddings-v2-base-de). The RAG selection mechanism is the one intervention that consistently improves LLM predictions in high-data settings, and the pseudonymization ablation shows the models rely on genuine category knowledge rather than surface patterns.

What would settle it

Run the SPWSLE classification using the original annotation mapping (or a mapping confirmed by the dataset creators) instead of the paper's estimated mapping; if the best LLM with RAG-selected shots then reaches or passes the BoW+SVM macro F1 of 0.761, the claim that LLMs cannot match extractive baselines on this task would be overturned. For the essay-grading claim, a concrete test would be to collect a larger corpus of graded German legal essays with multiple annotators and check whether any open pre-trained LLM produces a Spearman correlation with grades significantly above zero.

Watch

Extended reading notes

Core claim

The paper's central claim is that open-source pre-trained LLMs have enough instruction capability and German legal background knowledge for some legal-analysis tasks, but not for the tasks that legal education depends on. On the joint Gutachtenstil component classification of SPWSLE student case solutions, the best tested LLM (Llama 3 with ten RAG-selected example shots) reaches a macro F1 of 0.580 and accuracy of 0.710, while a Bag-of-Words linear SVM baseline reaches 0.761 and 0.784; the LLMs fall particularly short on the subsumption-related categories (Premise, Legal Claim, Subsumption). On the new Graded Strafrecht Hausarbeiten dataset, where full student legal opinions must be graded on a 0-18 scale, the models produce predictions no better than random guessing, even with extended context, simplified grading scales, and chain-of-thought prompts. The authors interpret this as a limitation of instruction following and German-language proficiency in current open models, and they show that retrieval-augmented selection of prompt examples is the most effective mitigation, while Chain-of-Thought helps only in the zero-shot case.

Load-bearing premise

The headline negative result about the German appraisal-style task rests on the authors' guess about how the original dataset labels map to the seven Gutachtenstil categories; the original paper does not document this mapping, so a different mapping would change the comparison.

Editorial extensions

If this is right

  • For Gutachtenstil component classification, practitioners should expect open pre-trained LLMs to underperform a cheap Bag-of-Words+SVM baseline even with retrieved examples, so the extractive baseline should not be discarded.
  • On the simpler tasks (CIMT public-feedback argument mining and English ASAP essay scoring), open LLMs with a few example shots match or beat the BoW baseline, and Chain-of-Thought prompting helps in the zero-shot case, making them usable in low-label regimes.
  • RAG-based example selection is a strong lever: choosing similar examples changes macro F1 by 14.4% on SPWSLE, and inverse retrieval drops performance below random shot selection, so prompt-example quality should be treated as a first-class design variable.
  • Full legal essay grading on the new GSHA dataset is beyond current open pre-trained LLMs; achieving it will require either substantially more capable models, fine-tuning on much larger graded corpora, or a different task formulation.
  • Even in the easier two-tiered extraction setting, LLMs trail the BERT-based and BoW classifiers, indicating the difficulty is inherent to component recognition rather than the joint evaluation scheme.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the original SPWSLE annotation mapping differs from the authors' estimate, the headline negative result would need recomputation; a direct follow-up is to obtain the documented mapping from the dataset authors and rerun both baseline and LLM conditions.
  • The authors note in the Limitations section that case-to-annotation linking is not present in the documents, so knowledge transfer to new within-domain tasks was untested; a concrete extension is to annotate a second set of legal cases and measure whether RAG-selected shots from one case still help on another.
  • The random-guessing result on GSHA could partly reflect annotation noise and the small 76-document corpus rather than model incapacity; a testable extension is to obtain multiple independent grades for the same essays and compare the model's rank correlation with the human inter-annotator correlation.
  • A practical hybrid suggested by the paper's numbers is to keep the BoW+SVM classifier for Gutachtenstil component extraction and reserve RAG-prompted LLMs for zero-shot or low-label settings, rather than replacing the baseline outright.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper evaluates the suitability of pre-trained generative LLMs (Llama 3 70B Instruct, Mixtral 8x7B Instruct, GPT-3.5 Turbo) for German legal analysis in educational contexts. It uses four datasets: two German argument-mining corpora (SPWSLE, CIMT) and two essay-scoring collections (a newly introduced Graded Strafrecht Hausarbeiten, GSHA, and ASAP set 8). The main results are that LLMs underperform a BoW+SVM baseline on SPWSLE Gutachtenstil classification (Table 3: BoW macro F1 0.761 vs. best LLM 0.580), perform near random on GSHA, but are competitive or better on CIMT and ASAP. The paper also introduces a RAG-based prompt example selection method that improves few-shot performance, evaluates Chain-of-Thought prompting, and reports a pseudonymization control experiment.

Significance. The paper addresses a timely and practically important question: can open-weight generative LLMs handle structured German legal analysis in education? If the negative results are robust, they provide a useful counterpoint to uncritical claims of LLM competence in specialized domains. The introduction of the GSHA dataset is a concrete contribution, the RAG-based example selection is systematically studied, and the pseudonymization experiment (Table 5) is a clever control for internalized category knowledge. However, the headline SPWSLE result rests on an explicitly estimated label mapping and a single data split, and the abstract's low-data superiority claim is not supported by reported baselines. The significance is therefore conditional on these methodological points being resolved.

major comments (4)
  1. [§4.2 (Evaluation Methodology) and Tables 2, 3, 9–12] The SPWSLE comparison relies on an estimated mapping from the original annotation scheme to the seven Gutachtenstil categories. The text states, "It is not exactly clear how they do this... Our estimate is the following: e1=Major Claim, e2=Conclusion, e4=Definition, e5=Subsumption, e6=Legal Claim, e7=Premise, with e3 remaining unused throughout all data." Every SPWSLE result, including the headline gap (BoW+SVM macro F1 0.761 vs. Llama 3 10RAG 0.580 in Table 3), is computed on this reconstruction. If e3 is used or e5/e6/e7 are permuted relative to the true scheme, the class-level F1 values and the overall gap would measure a different task. Please verify the mapping by contacting the original authors or inspecting annotation guidelines, or report a sensitivity analysis over plausible alternative mappings. At minimum, the caveat should be stated prominently wherever "cannot match" is claimed.
  2. [Abstract and §4.3, Table 4] The abstract claims, "Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case." No BoW+SVM baseline is reported for the low-data scenarios. Table 4 compares only Llama 3 variants (CoT vs. result-only) at 80%, 0.3%, and 0% training data, and Tables 6–7 always use the full training set for the BoW baseline. The "throughout" claim is therefore unsupported. Please add low-data BoW baselines (e.g., training the SVM on the same subsamples) or remove/qualify the claim to the specific tasks where it actually holds.
  3. [§4.3, Tables 2–3 and 9] All SPWSLE results are computed on a single 20% test split, with no variance estimates or cross-validation. Given the dataset size (382 documents) and the strong negative conclusion ("cannot match" in §4.3), the results could depend on the idiosyncrasies of one split. Please report k-fold cross-validation or repeated random splits with standard deviations for the main comparison, at least for the BoW+SVM baseline and the best LLM configuration.
  4. [§4.5, Figure 6] The claim that LLMs "cannot achieve prediction quality beyond random guessing" on GSHA is not supported by reproducible numbers. Figure 6 is referenced but no quantitative correlation/accuracy values or a formal random baseline are given in the text. Please report the actual Spearman/Pearson correlations and accuracies with standard errors, and define what "random guessing" means for this 0–18 grading scale (e.g., majority-class or uniform random prediction).
minor comments (7)
  1. [§4.3.2] The phrase "not no par" appears to be a typo for "not on par."
  2. [Table 11 caption] The word "Detauled" should be "Detailed."
  3. [Table 10] Both rows in Table 10 are labeled "RAG" in the table body, but the caption indicates that the second row is "Inverse RAG." Please correct the row labels for clarity.
  4. [§4.1.2 and Appendix] Since GSHA is a newly introduced dataset containing student work, please specify the anonymization/pseudonymization procedure and the intended data-release license, in addition to the consent statement already present.
  5. [§3 Methods, Table 8] The LiveBench scores in Table 8 are presented but never explicitly used in the analysis. Either connect them to the interpretation of the German-task results or remove them to tighten the exposition.
  6. [§3.5 and Table 4] The Generated Artificial Rationales (GAR) method is introduced as a contribution, but Table 4 shows it never improves over the result-only prompt and in some cases degrades performance. Consider framing GAR explicitly as a negative result rather than part of the proposed method.
  7. [Figure 6] Figure 6 is mentioned in §4.5 but does not appear in the text extract; ensure the final version includes the figure and that its axes and error bars are legible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark evaluation against external datasets, and the same-team citation is motivational, not load-bearing.

full rationale

The central claims are derived from direct evaluations on four external benchmark datasets (SPWSLE, CIMT, ASAP, GSHA) against fixed baselines (BoW+SVM, BERT from Weber et al. 2023), with no fitted parameter or learned component feeding back into the headline comparison; Tables 2-7 and 9-12 are measured outputs of the same prompting and evaluation protocol. The SPWSLE label mapping in Section 4.2 is explicitly an estimate rather than a derived result: the paper states 'It is not exactly clear how they do this' and 'there is no mapping of annotation categories to classes included,' then proposes a mapping, but this mapping is an input assumption, not a conclusion forced by the data, and it is applied identically to all compared methods. The only same-team citation (Al Zubaer et al. 2023) appears in Related Work and motivates example-based prompting; the paper's own RAG experiments on SPWSLE, CIMT, and ASAP provide independent evidence for that motivation, so the citation is not load-bearing. The pseudonymization and chain-of-thought ablations are internal consistency checks rather than derived predictions, and the GSHA 'beyond random guessing' statement is a direct correlation result. The paper also discloses limitations (e.g., unreproducible Romberg and Conrad SVM features, missing case linking), which are transparency concerns, not circular reasoning. No prediction in the paper reduces by construction to a fitted input or to a self-citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The paper's headline comparisons rest on task operationalization and prompt interpretation choices rather than on fitted numerical constants. The key hidden costs are the estimated SPWSLE label mapping, the pattern-matching readout, and the unverified reliability of single-annotator GSHA grades.

free parameters (3)
  • Number of example shots (k=10) = 10
    Hand-chosen across all few-shot experiments; no sweep is reported, and all comparisons of LLMs to baselines use this fixed prompt size.
  • RAG retrieval top-k = 10
    Same hand-selected k=10 for retrieval-augmented prompt examples; no tuning curve is given.
  • CoT explanation length cap = 100 words
    The CoT prompt explicitly asks for up to 100 words of German explanation, an arbitrary constraint that can affect output length and classification readout.
assumptions (4)
  • domain assumption The estimated SPWSLE class mapping (e1=Major Claim, e2=Conclusion, e4=Definition, e5=Subsumption, e6=Legal Claim, e7=Premise) is correct.
    Used to compute all SPWSLE results; the authors state the original mapping is unclear in Weber et al. (2023), Section 4.2.
  • domain assumption The joint one-pass classification task captures Gutachtenstil recognition ability.
    The paper replaces the original two-tiered extraction with a joint task and treats it as the main evidence for LLM breakdown, Section 4.2.
  • ad hoc to paper Pattern matching of the most frequent category in the response is a valid readout of LLM predictions.
    Used in Section 4.3.2 because CoT responses are malformed; assumes category mentions in free text reliably reflect model classification.
  • domain assumption The single-annotator grades in GSHA are reliable enough to measure scoring accuracy.
    Section 4.2 replaces QWK with correlations because each essay has one grade; the small 76-document dataset underpins the near-random-grading conclusion.
invented entities (1)
  • GSHA dataset (Graded Strafrecht Hausarbeiten)
    purpose: New benchmark of 76 graded German criminal-law essays used to conclude that pre-trained LLMs cannot score full legal essays.
    The dataset is introduced in Section 4.1.2 but no release link, license, or anonymization details are provided, so other groups cannot yet check or reuse it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education." pith.science (2026). https://pith.science/paper/AOHGHEWQ

@misc{pith2026241215902,
  author       = {Pith},
  title        = {Pith review of: On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AOHGHEWQ}},
  note         = {Machine review of arXiv:2412.15902}
}
read the original abstract

We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case.

Figures

Figures reproduced from arXiv: 2412.15902 by the authors.

Figure 1
Figure 1. Spearman ranking correlation for the ASAP AES transfer study, mean over 3 folds. siderably in linear correlation and CoT does not offer any improvement. This suggests that even with the slightly more complex set 8 we are observing the limits of model capability. The improvements over the baseline with Llama 310 are still a good sign for LLM-based AES in education settings. Model Spearman Pearson BoW+SVM .586±.04 .61… view at source ↗
Figure 4
Figure 4. Chain-of-Thought prompt for SPWSLE Gutachtenstil argument mining [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 3
Figure 3. Translated result prompt for SPWSLE Gutachtenstil argument mining. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information pro￾cessing systems, 35:24824–24837. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Nee… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Mean proportion of shots with the cor￾rect label for a request in SPWSLE Gutachtenstil argument mining. Random selection vs. RAG￾based selection via cosine similarity over sentence embeddings [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: GSHA automatic essay scoring results with few-shot prompting in 5-fold cross-validation repeated 3 times [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages

  1. [4]

    B Hamner, J Morgan, MS Lynnvandev, and TV Ark

    Can large language models perform relation-based argument mining? Preprint, arXiv:2402.11243. B Hamner, J Morgan, MS Lynnvandev, and TV Ark

    arXiv

  2. [5]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al

    Large lan- guage models in law: A survey.arXiv preprint arXiv:2312.03718. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al

    arXiv

  3. [6]

    arXiv preprint arXiv:2402.17016

    Multi-task contrastive learn- ing for 8192-token bilingual text embeddings. arXiv preprint arXiv:2402.17016. Christopher M Ormerod, Akanksha Malhotra, and Amir Jafari

    arXiv

  4. [8]

    Explor- ing large language models in financial argument relation identification. In Proceedings of the Joint Workshop of the 7th Financial Technol- ogy and Natural Language Processing, the 5th Knowledge Discovery from Unstructured Data in Financial Services, and the 4th Workshop on Economics and Natural Language Processing @ LREC-COLING 2024, pages 119–12...

    work page 2024

  5. [10]

    arXiv preprint arXiv:2402.07927

    A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927. Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth

    arXiv

  6. [11]

    arXiv preprint arXiv:2404.15845

    Exploring llm prompting strategies for joint essay scor- ing and feedback generation. arXiv preprint arXiv:2404.15845. Kaveh Taghipour and Hwee Tou Ng

    arXiv

  7. [13]

    InFindings of the Association for Com- putational Linguistics: ACL 2023, pages 2296–

    Struc- tured persuasive writing support in legal educa- tion: A model and tool for german legal case so- lutions. InFindings of the Association for Com- putational Linguistics: ACL 2023, pages 2296–

    work page 2023

  8. [14]

    arXiv preprint arXiv:2401.06431

    From automation to augmentation: Large lan- guage models elevating essay scoring landscape. arXiv preprint arXiv:2401.06431. Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He

    arXiv

Show all 16 references
  1. [15]

    InFind- ings of the Association for Computational Lin- guistics: EMNLP 2020, pages 1560–1569

    En- hancing automated essay scoring performance viafine-tuningpre-trainedlanguagemodelswith combination of regression and ranking. InFind- ings of the Association for Computational Lin- guistics: EMNLP 2020, pages 1560–1569. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola

    2020

  2. [2016]

    In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1882–1891

    A neu- ral approach to automated essay scoring. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1882–1891. Florian Weber, Thiemo Wambsganss, Seyed Parsa Neshaei, and Matthias Soellner

    2016

  3. [2019]

    arXiv preprint arXiv:1909.09482

    Language models and automated essay scoring. arXiv preprint arXiv:1909.09482. Julia Romberg and Stefan Conrad

    1909 arXiv

  4. [2020]

    Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera Lú- cia Raposo, Sofia Morgado, et al

    Language mod- els are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera Lú- cia Raposo, Sofia Morgado, et al

    1901

  5. [2021]

    arXiv preprint arXiv:2102.13136

    Automated essay scoring us- ing efficient transformer-based language models. arXiv preprint arXiv:2102.13136. Yasser Otiefy and Alaa Alhamzeh

    arXiv

  6. [2022]

    arXiv preprint arXiv:2210.03493

    Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493. A Appendix Text: <TEXT> Explain: What part of the Gutachtenstil is this? Briefly explain your decision in German, up to 100 words. Figure 4: Chain-of-Thought prompt for SPWSLE Gutac...

    arXiv

  7. [2023]

    arXiv preprint arXiv:2305.13245

    Gqa: Training generalized multi- query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245. Abdullah Al Zubaer, Michael Granitzer, and Je- lena Mitrović

    arXiv

  8. [2024]

    arXiv preprint arXiv:2403.03883

    Saullm- 7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883. Deniz Gorur, Antonio Rago, and Francesca Toni

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.