Pith. sign in

REVIEW 3 major objections 4 minor 95 references

This paper argues that domain-specific data and architecture matter more than parameter count: a 28-billion-parameter tax-trained model beats general-purpose models of comparable and much larger size on authentic German tax-law exams.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:12 UTC pith:JWOZKRHZ

load-bearing objection SteuerEx is a real asset for legal NLP, but the paper's headline comparison is unsupported because the training seed and the benchmark come from the same FAU exam pool and no holdout is shown. the 3 major comments →

arxiv 2602.11081 v1 pith:JWOZKRHZ submitted 2026-02-11 cs.CL cs.AIcs.LG

SteuerLLM: Local specialized large language model for German tax law analysis

Sebastian Wind , Jeta Sopa , Laurin Schmid , Quirin Jackl , Sebastian Kiefer , Fei Wu , Martin Mayr , Harald K\"ostler
show 3 more authors
Gerhard Wellein Andreas Maier Soroosh Tayebi Arasteh
This is my paper
classification cs.CL cs.AIcs.LG
keywords German tax lawSteuerExSteuerLLMlegal reasoning benchmarkdomain-adapted LLMsynthetic training datapartial-credit evaluationblock expansion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a 28-billion-parameter model trained specifically on German tax law can outperform general-purpose instruction-tuned models of similar and much larger size on realistic German tax-law examinations. To test this, it introduces SteuerEx, an open benchmark of 115 authentic university exam questions scored by decomposing expert reference solutions into point-weighted legal statements and awarding partial credit. It also builds SteuerLLM on top of a 24-billion-parameter base by inserting extra transformer layers (block expansion) and training on roughly 485,000 synthetic question-answer pairs generated from authentic exam material and legal sources through a retrieval-augmented pipeline. On the benchmark, SteuerLLM scores above all general-purpose models tested except the largest 671-billion-parameter reasoning system, while students still outperform all models on average. The intended upshot is that for codified, high-stakes domains, training composition and architectural adaptation can matter more than raw scaling.

Core claim

On its own terms, the paper's central discovery is that SteuerLLM, a 28B-parameter model built by freezing a pretrained 24B Transformer and inserting eight extra trainable layers, reaches 28% on the SteuerEx benchmark, while general-purpose instruction-tuned models of 32B, 72B, and even a 70B reasoning-distilled model remain in the 18–20% range, and a proprietary mini-model reaches only 22%. Only a 671B reasoning-oriented model scores higher. The largest gains appear in corporate tax and fundamentals of tax law, where SteuerLLM matches a much larger model in corporate tax; the gains are smaller and noisier in narrowly sampled categories with few questions. The paper attributes the result to

What carries the argument

The argument rests on three linked mechanisms. First, the Water Fountain Algorithm: starting from roughly 1,800 authentic exam questions, it retrieves legal sources, assembles bounded contexts by semantic similarity, generates answers with an explicit insufficiency flag, and multiplies each accepted question into three new variants, yielding 485,092 validated question-answer pairs after cleansing. Second, block expansion: eight new Transformer layers are interleaved into a frozen 24B pretrained network, initialized so the residual pathway is near-identity, so domain specialization is added without overwriting general abilities. Third, statement-level partial-credit evaluation: each reference

Load-bearing premise

The load-bearing premise is that the 115 benchmark exam questions were not included among the roughly 1,800 seed questions that generated the synthetic training data, since both come from the same university examination pool; if that premise fails, the performance gap may reflect memorization rather than domain-adapted reasoning.

What would settle it

Search the released SteuerEx benchmark and released training data for exact or near-duplicate question texts. If a benchmark question appears in the roughly 1,800-question seed set or in generated training pairs, the central claim is falsified; if no overlap appears, the claim survives this check.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Moderately sized, locally runnable models can compete with much larger systems on structured legal reasoning, lowering the practical cost of deploying specialized legal AI.
  • A fully open pipeline—synthetic data, model weights, evaluation code—can reproduce most of the performance gain, so the approach does not depend on a private data asset.
  • Authentic exam material with partial-credit grading exposes capability differences that conventional legal NLP benchmarks miss, making it a more faithful evaluation format for legal reasoning.
  • General scaling laws from language benchmarks do not transfer automatically to formally constrained domains; training-data composition and alignment with answer structure dominate in this setting.
  • The persistent gap to average student performance shows the same exam format can serve as a continuing diagnostic for legal AI progress.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the released benchmark and the roughly 1,800-question seed set are cross-checked and overlap is found, the 'domain generalization' reading of the results would need to be replaced by a 'training-distribution fit' reading; the paper does not currently rule this out.
  • Because the evaluation is deliberately closed-book, a natural extension—also suggested by the paper's framing—is to add retrieval grounding at answer time, which could close part of the gap to human students.
  • The low agreement among human graders on individual statements implies benchmark scores are best treated as relative comparisons under one fixed automated rubric, not as absolute measures of legal correctness; future work could report both automated and human-variance bands.
  • The same block-expansion plus synthetic-data recipe is a testable template for other codified, rule-heavy domains such as accounting standards, customs law, or regulatory compliance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces two linked contributions. First, SteuerEx, a benchmark of 115 expert-validated German tax-law examination questions from FAU, scored at statement level with partial credit. Second, SteuerLLM, a 28B-parameter model built by inserting trainable blocks into Mistral Small 2409 while freezing the original layers, trained on roughly 485,000 synthetic question–answer pairs generated by the "Water Fountain Algorithm" from about 1,800 authentic FAU exam questions. The authors evaluate 17 models, report that SteuerLLM outperforms all general-purpose baselines except DeepSeek-R1-671B, and interpret this as evidence that domain-specific data and architectural adaptation matter more than parameter scale. They also report a human-evaluation study of the automated grader and an open-weight variant, Open-SteuerLLM.

Significance. If the central claim is valid, the paper would be a useful contribution: SteuerEx is a realistic, expert-validated benchmark that is more faithful to actual legal education than existing legal NLP benchmarks, and the partial-credit statement-level evaluation is a sensible design. The authors also do several things right: they release the benchmark, the open model, and the synthetic training data; they validate the automated evaluator against human judgments; and they transparently discuss the subjectivity of legal grading (ICC(2,1)=0.367 while Kendall's tau between averaged human scores and the LLM evaluator is 0.718). These are genuine strengths. However, the headline comparative claim depends entirely on SteuerEx being an unseen test set, and the paper does not establish this. The score inconsistency between Table 2 and Supplementary Table 6 further obscures what the primary reported metric actually is. The paper's contribution is therefore currently conditional on an unverified holdout assumption.

major comments (3)
  1. [Methods, "Algorithmic generation of the training dataset" and "SteuerEx benchmark"] The central claim that SteuerLLM beats much larger models requires SteuerEx to be out-of-distribution for training. The Water Fountain seed is "approximately 1,800 authentic tax law examination questions from FAU" and SteuerEx is built from "authentic university tax law examinations administered at FAU," with overlapping semesters (WS16/17 through WS23/24). The data-cleaning step removes overlaps "with the initial seed dataset," but never states that the 115 benchmark questions were excluded from the seed or from the generated pool. Code availability explicitly withholds the evaluation pipeline to avoid "benchmark leakage," yet no analogous holdout guarantee is given for the training data. If benchmark questions or close variants appear in the seed or in the 485,092 generated pairs, the 28% vs. 19-23% gap could reflect memorization, not domain-adaptive generalization. The authors must re
  2. [Table 2 vs. Supplementary Table 6] There is a direct numerical inconsistency in the headline scores. Table 2 reports SteuerLLM at 28 +/- 2 (294/1035.5 points), while Supplementary Table 6 reports observed accuracy 32.93% and bootstrap mean accuracy 28.38%. The 294-point value corresponds to the bootstrap mean, not the observed total (which would be about 341 points). The same pattern appears for DeepSeek-R1-671B (39 vs. 44.19), GPT-4o-mini (22 vs. 26.90), and other models. If the primary scores in Table 2 are points-constrained bootstrap means, the table's "Total points" column and the definition in Eq. (13) must be corrected, and the relation between these bootstrap means and the permutation-based p-values must be explained. As written, the abstract and Results report numbers that are not the observed benchmark scores.
  3. [Methods, "Statistical analysis" and Supplementary Note 1] The points-constrained bootstrap is acknowledged to change question inclusion probabilities and to produce shifts of up to -6.7 percentage points relative to observed accuracy. Since Table 2 apparently reports these bootstrap means as the main metric while the p-values are computed from paired permutation tests on observed per-question differences, the two sets of inferential statements target different estimands. This is not merely a presentation issue: it can change pairwise conclusions (e.g., Open-SteuerLLM vs. SteuerLLM) and should be resolved by reporting observed points-weighted accuracy as the primary metric and treating the constrained bootstrap as a sensitivity analysis.
minor comments (4)
  1. [Results, paragraph 1] Typo: "exteremly" should be "extremely."
  2. [Table 4 vs. Table 3] SteuerLLM category grades in Table 4 differ from the corresponding rows in Table 3 (e.g., corporate tax 36.2% vs. 38%, fundamentals 49.2% vs. 41%, VAT 32.5% vs. 35%). If the student-comparison normalization differs from the benchmark normalization, this should be stated explicitly; at present the reader cannot reconcile the two tables.
  3. [Code availability] The text says "All source code, configurations, parameters, models, and workflows used in this study are publicly available" and then states the evaluation pipeline is withheld. Please state precisely which components are released immediately and which are deferred, and avoid the blanket sentence.
  4. [Human evaluation] The human-LLM agreement is computed on only 59 valid statement-level items after excluding 4 malformed outputs; the per-model Kendall tau CIs are wide (e.g., 0.38-0.96). The claim that the evaluator "does not favor a particular architecture" is stronger than the sample supports. This should be softened or supported with additional data.

Circularity Check

1 steps flagged

SteuerEx and the Water Fountain training seed both draw from the same FAU exam pool, and the paper never documents benchmark holdout from the seed or the synthetic training set, making SteuerLLM's headline advantage partially circular.

specific steps
  1. other [Methods, 'Algorithmic generation of the training dataset'; Methods, 'SteuerEx benchmark'; 'Experimental design'; Code availability]
    "The algorithm is initialized with a curated seed dataset consisting of approximately 1,800 authentic tax law examination questions from FAU. ... The dataset originates from original tax law examinations conducted by the Chair of Tax Law and Public Law at FAU. ... All evaluation questions in this study are drawn from authentic German university tax law examinations administered ... at FAU. ... To mitigate risks of training data contamination and benchmark leakage, the LLM-based evaluation pipeline is not released publicly at this time."

    SteuerLLM's central result is measured on SteuerEx, 115 questions from authentic FAU tax-law examinations. The same source pool—authentic FAU examination questions—is the seed for the Water Fountain Algorithm that generated the 485,092 training pairs; the algorithm even adds each accepted question q into the dataset D. The paper says cleansing removed duplicates/overlaps with the seed, but it never states that the 115 benchmark questions were excluded from the seed, from D, or from the context-supported regeneration component. The only explicit leakage-mitigation statement concerns the grading pipeline, not the training data. As written, therefore, the benchmark can be a part of the distribution on which SteuerLLM was optimized, so the reported advantage over general-purpose models is not

full rationale

The paper has no fitted equation masquerading as a derivation, and no load-bearing self-citation of the kind described in the rubric. The central circularity risk is the unresolved overlap between the benchmark and the training-generation seed. Both SteuerEx and the Water Fountain seed are defined as authentic FAU tax-law examination questions covering the same institution and overlapping semesters; the authorial text describing the training algorithm adds each seed question q into D, and the cleansing stage only removes overlaps with the seed without documenting that the 115 benchmark items were held out. Since the headline claim—that domain-specific data and architectural adaptation beat parameter scale—depends entirely on SteuerEx being an independent test set, the absence of any stated disjointness makes the evaluation loop partially self-referential. I am not asserting that overlap definitely occurred, but under the reviewing rule the paper itself supplies the relevant premises and never supplies the missing holdout support; the burden is on the paper to establish test-set independence. The Table 2 versus Supplementary Table 6 score inconsistency (e.g., SteuerLLM total points 294.0 vs observed accuracy 32.93%) is a serious correctness issue but is not itself circularity. No uniqueness theorem, ansatz-importing citation, or renaming of a known result is present. Because the central comparative claim is not shown to be independent of the training-input distribution, a partial-circularity score of 6 is warranted; if explicit disjointness were documented, the score would drop to 0–2.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new physical entities are postulated; the Water Fountain Algorithm is a data-generation procedure, not an invented entity. The main burden is the unstated benchmark/training disjointness assumption and hand-chosen generation thresholds that shape the training distribution.

free parameters (5)
  • Water Fountain growth factor k = 3
    Equation (6); chosen by hand; controls exponential dataset expansion and therefore the training distribution.
  • Minimum retrieved sources S_min = 3
    Data-cleansing threshold; tuples with fewer than 3 independent sources are discarded. Hand-chosen.
  • Context token limit N = model context window (e.g., 32k)
    Equation (2); bounds context construction during synthetic data generation and influences answer quality.
  • Number of inserted transformer blocks = 8 (24B to 28B)
    Architecture choice for SteuerLLM; determines the added domain-specific capacity.
  • Training token/pair budgets = 6.3B pretraining tokens; ~1.5M instruction pairs
    Compute-budget choices that shape the final model and cannot be independently justified.
axioms (4)
  • domain assumption FAU authentic exams adequately represent German tax-law reasoning
    Benchmark and training seed come from one institution; authors acknowledge limited cross-institutional generalization.
  • domain assumption LLM-generated synthetic Q&A pairs are legally accurate enough for training
    No systematic legal verification beyond an expert-defined taxonomy, algorithmic filtering, and source-count threshold.
  • domain assumption GPT-4o statement-level scoring is a valid proxy for expert grading
    Supported by Kendall tau 0.718, but human inter-rater ICC is only 0.367 and partial-credit grading is acknowledged to be subjective.
  • ad hoc to paper SteuerEx benchmark questions are disjoint from SteuerLLM training data
    Unstated but required for the comparative claim; the Water Fountain seed of ~1,800 FAU exam questions overlaps the benchmark source pool.

pith-pipeline@v1.3.0-alltime-deepseek · 33136 in / 12166 out tokens · 122073 ms · 2026-08-03T00:12:19.453743+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SteuerLLM: Local specialized large language model for German tax law analysis." pith.science (2026). https://pith.science/paper/JWOZKRHZ

@misc{pith2026260211081,
  author       = {Pith},
  title        = {Pith review of: SteuerLLM: Local specialized large language model for German tax law analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JWOZKRHZ}},
  note         = {Machine review of arXiv:2602.11081}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) demonstrate strong general reasoning and language understanding, yet their performance degrades in domains governed by strict formal rules, precise terminology, and legally binding structure. Tax law exemplifies these challenges, as correct answers require exact statutory citation, structured legal argumentation, and numerical accuracy under rigid grading schemes. We algorithmically generate SteuerEx, the first open benchmark derived from authentic German university tax law examinations. SteuerEx comprises 115 expert-validated examination questions spanning six core tax law domains and multiple academic levels, and employs a statement-level, partial-credit evaluation framework that closely mirrors real examination practice. We further present SteuerLLM, a domain-adapted LLM for German tax law trained on a large-scale synthetic dataset generated from authentic examination material using a controlled retrieval-augmented pipeline. SteuerLLM (28B parameters) consistently outperforms general-purpose instruction-tuned models of comparable size and, in several cases, substantially larger systems, demonstrating that domain-specific data and architectural adaptation are more decisive than parameter scale for performance on realistic legal reasoning tasks. All benchmark data, training datasets, model weights, and evaluation code are released openly to support reproducible research in domain-specific legal artificial intelligence. A web-based demo of SteuerLLM is available at https://steuerllm.i5.ai.fau.de.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

95 extracted references · 22 linked inside Pith

  1. [1]

    Brown, T. B. et al. Language models are few-shot learners. in Proceedings of the 34th International Conference on Neural Information Processing Systems vol. 159 1877–1901 (2020)

    1901

  2. [2]

    Vaswani, A. et al. Attention Is All You Need. in NIPS’17: Proceedings of the 31st International Conference on Neural Information Processing Systems 6000–6010 (2017)

    2017

  3. [3]

    Wind, S. et al. Multi-step retrieval and reasoning improves radiology question answering with large language models. npj Digit. Med. 8, 790 (2025)

    2025

  4. [4]

    Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837 (2022)

    2022

  5. [5]

    & Van Durme, B

    Blair-Stanek, A., Holzenberger, N. & Van Durme, B. Can gpt-3 perform statutory reasoning? in Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law 22–31 (2023)

    2023

  6. [6]

    & Altwicker, T

    Dugac, G. & Altwicker, T. Classifying legal interpretations using large language models. Artificial Intelligence and Law 1–19 (2025)

    2025

  7. [7]

    Dietterich, T. G. Steps toward robust artificial intelligence. Ai Magazine 38, 3–24 (2017)

    2017

  8. [8]

    Wei, J. et al. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)

    Pith/arXiv arXiv 2021

  9. [9]

    The next decade in AI: four steps towards robust artificial intelligence

    Marcus, G. The next decade in AI: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177 (2020)

    Pith/arXiv arXiv 2002

  10. [10]

    Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. in Advances in Neural Information Processing Systems (eds Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H.) vol. 33 9459–9474 (Curran Associates, Inc., 2020)

    2020

  11. [11]

    Chalkidis, I. et al. LexGLUE: A benchmark dataset for legal language understanding in English. in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 4310–4330 (2022)

    2022

  12. [12]

    & Androutsopoulos, I

    Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N. & Androutsopoulos, I. LEGAL- BERT: The muppets straight out of law school. arXiv preprint arXiv:2010.02559 (2020)

    Pith/arXiv arXiv 2010

  13. [13]

    & Zhu, X

    Li, J., Bhambhoria, R. & Zhu, X. Parameter-efficient legal domain adaptation. arXiv preprint arXiv:2210.13712 (2022)

    Pith/arXiv arXiv 2022

  14. [14]

    Colombo, P. et al. Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain. Advances in Neural Information Processing Systems 37, 129672–129695 (2024)

    2024

  15. [15]

    Hendrycks, D. et al. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020)

    Pith/arXiv arXiv 2009

  16. [16]

    Rabelo, J. et al. Overview and discussion of the competition on legal information extraction/entailment (COLIEE) 2021. The Review of Socionetwork Strategies 16, 111–133 (2022)

    2021

  17. [17]

    & Berteloot, P

    Bernet, H. & Berteloot, P. EUR-Lex: A multilingual on-line website for European Union law. International Review of Law Computers & Technology 20, 337–339 (2006)

    2006

  18. [18]

    & Gertz, M

    Aumiller, D., Chouhan, A. & Gertz, M. EUR-lex-sum: A multi-and cross-lingual dataset for long-form summarization in the legal domain. arXiv preprint arXiv:2210.13448 (2022)

    Pith/arXiv arXiv 2022

  19. [19]

    Einkommensteuergesetz: Kommentar

    Kirchhof, P. Einkommensteuergesetz: Kommentar. (Verlag Dr. Otto Schmidt, 2012)

    2012

  20. [20]

    & Tipke, K

    Lang, J. & Tipke, K. Steuerrecht. (Otto Schmidt, 2020)

    2020

  21. [21]

    & Gelter, M

    Grechenig, K. & Gelter, M. The transatlantic divergence in legal thought: American law and economics vs. German doctrinalism. Hastings Int’l & Comp. L. Rev. 31, 295 (2008)

    2008

  22. [22]

    McGinnis, J. O. & Pearce, R. G. The great disruption: How machine intelligence will transform the role of lawyers in the delivery of legal services. Fordham L. Rev. 82, 3041 (2013)

    2013

  23. [23]

    Wang, A. et al. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. in Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and 37 Interpreting Neural Networks for NLP 353–355 (Association for Computational Linguistics, Brussels, Belgium, 2018). doi:10.18653/v1/W18-5446

    doi:10.18653/v1/w18-5446 2018

  24. [24]

    Son, G. et al. KMMLU: Measuring Massive Multitask Language Understanding in Korean. in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) 4076–4104 (Association for Computational Linguistics, Albuquerque, New Mexico, 2025). doi:10.1...

    doi:10.18653/v1/2025.naacl-long.206 2025

  25. [25]

    & Liang, P

    Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing 2383–2392 (Association for Computational Linguistics, Austin, Texas, 2016). doi:10.18653/v1/D16-1264

    doi:10.18653/v1/d16-1264 2016

  26. [26]

    Wolfson, T. et al. Break It Down: A Question Understanding Benchmark. Transactions of the Association for Computational Linguistics 8, 183–198 (2020)

    2020

  27. [27]

    Kaplan, J. et al. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    Pith/arXiv arXiv 2001

  28. [28]

    Hoffmann, J. et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022)

    Pith/arXiv arXiv 2022

  29. [29]

    Zheng, L. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, 46595–46623 (2023)

    2023

  30. [30]

    & Schütze, H

    Schick, T. & Schütze, H. Generating datasets with pretrained language models. arXiv preprint arXiv:2104.07540 (2021)

    Pith/arXiv arXiv 2021

  31. [31]

    Eldan, R. & Li, Y. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759 (2023)

    Pith/arXiv arXiv 2023

  32. [32]

    M., Bommarito II, M

    Katz, D. M., Bommarito II, M. J. & Blackman, J. A general approach for predicting the behavior of the Supreme Court of the United States. PloS one 12, e0174698 (2017)

    2017

  33. [33]

    Katz, D. M. & Bommarito, M. J. Measuring the complexity of the law: the United States Code. Artificial intelligence and law 22, 337–374 (2014)

    2014

  34. [34]

    Machine learning and law

    Surden, H. Machine learning and law. Wash. L. Rev. 89, 87 (2014)

    2014

  35. [35]

    & Chang, M

    Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M. Retrieval augmented language model pre-training. in International conference on machine learning 3929–3938 (PMLR, 2020)

    2020

  36. [36]

    & Grave, E

    Izacard, G. & Grave, E. Leveraging passage retrieval with generative models for open domain question answering. in Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume 874–880 (2021)

    2021

  37. [37]

    Textual Spatial Cosine Similarity

    Crocetti, G. Textual Spatial Cosine Similarity. Preprint at https://doi.org/10.48550/arXiv.1505.03934 (2015)

    Pith/arXiv arXiv doi:10.48550/arxiv.1505.03934 2015

  38. [38]

    & Shafique, M

    Shao, M., Basit, A., Karri, R. & Shafique, M. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges. IEEE Access 12, 188664–188706 (2024)

    2024

  39. [39]

    & Weinberger, K

    Huang, G., Sun, Y., Liu, Z., Sedra, D. & Weinberger, K. Q. Deep Networks with Stochastic Depth. in Computer Vision – ECCV 2016 (eds Leibe, B., Matas, J., Sebe, N. & Welling, M.) vol. 9908 646–661 (Springer International Publishing, Cham, 2016)

    2016

  40. [40]

    & Sun, J

    He, K., Zhang, X., Ren, S. & Sun, J. Identity Mappings in Deep Residual Networks. in Computer Vision – ECCV 2016 (eds Leibe, B., Matas, J., Sebe, N. & Welling, M.) vol. 9908 630–645 (Springer International Publishing, Cham, 2016)

    2016

  41. [41]

    & Ruder, S

    Howard, J. & Ruder, S. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 (2018)

    Pith/arXiv arXiv 2018

  42. [42]

    Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. U.S.A. 114, 3521–3526 (2017)

    2017

  43. [43]

    & Jauvin, C

    Bengio, Y., Ducharme, R., Vincent, P. & Jauvin, C. A neural probabilistic language model. The Journal of Machine Learning Research 3, 1137–1155

  44. [44]

    & Hutter, F

    Loshchilov, I. & Hutter, F. Decoupled Weight Decay Regularization. in Proceedings of Proceedings of Seventh International Conference on Learning Representations (ICLR) 2019 (New Orleans, LA, USA, 2019). 38

    2019

  45. [45]

    Shoeybi, M. et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. Preprint at https://doi.org/10.48550/arXiv.1909.08053 (2020)

    Pith/arXiv arXiv doi:10.48550/arxiv.1909.08053 1909

  46. [46]

    Narayanan, D. et al. Efficient large-scale language model training on GPU clusters using megatron-LM. in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis 1–15 (ACM, St. Louis Missouri, 2021). doi:10.1145/3458817.3476209

    arXiv 2021

  47. [47]

    & Fehrenbacher, O

    Schnitger, A. & Fehrenbacher, O. Kommentar Körperschaftsteuer KStG. (Springer, 2012)

    2012

  48. [48]

    Lloyd, A. C. The Logical Form of Law Statements. Mind 64, 312–318 (1955)

    1955

  49. [49]

    Zhong, H. et al. How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence. in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 5218–5230 (Association for Computational Linguistics, Online, 2020). doi:10.18653/v1/2020.acl-main.466

    doi:10.18653/v1/2020.acl-main.466 2020

  50. [50]

    & Choi, Y

    Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. The Curious Case of Neural Text Degeneration. in ICLR 2020 (2020)

    2020

  51. [55]

    Team, G. et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)

    Pith/arXiv arXiv 2024

  52. [56]

    Team, G. et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  53. [57]

    (Shane), Reid, M., Matsuo, Y

    Kojima, T., Gu, S. (Shane), Reid, M., Matsuo, Y. & Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. in Advances in Neural Information Processing Systems (eds Koyejo, S. et al.) vol. 35 22199–22213 (Curran Associates, Inc., 2022)

    2022

  54. [58]

    GPT-4 Technical Report

    OpenAI. GPT-4 Technical Report. Preprint at http://arxiv.org/abs/2303.08774 (2023)

    Pith/arXiv arXiv 2023

  55. [59]

    & Eger, S

    Zhao, W., Strube, M. & Eger, S. DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence. in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics 3865–3883 (Association for Computational Linguistics, Dubrovnik, Croatia, 2023). doi:10.18653/v1/2023.eacl-main.278

    doi:10.18653/v1/2023.eacl-main.278 2023

  56. [60]

    Raji, I. D. et al. Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency 33–44 (ACM, Barcelona Spain, 2020). doi:10.1145/3351095.3372873

    arXiv 2020

  57. [61]

    & Steinhardt, J

    Hendrycks, D., Carlini, N., Schulman, J. & Steinhardt, J. Unsolved Problems in ML Safety. Preprint at https://doi.org/10.48550/arXiv.2109.13916 (2022)

    Pith/arXiv arXiv doi:10.48550/arxiv.2109.13916 2022

  58. [62]

    & Katz, D

    Bommarito, M. & Katz, D. M. GPT Takes the Bar Exam. Preprint at https://doi.org/10.48550/arXiv.2212.14402 (2022)

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.14402 2022

  59. [64]

    Shrout, P. E. & Fleiss, J. L. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 86, 420–428 (1979)

    1979

  60. [65]

    Kendall, M. G. A New Measure of Rank Correlation. Biometrika 30, 81 (1938)

    1938

  61. [66]

    Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit

    Cohen, J. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin 70, 213–220 (1968)

    1968

  62. [67]

    & Tibshirani, R

    Efron, B. & Tibshirani, R. J. An Introduction to the Bootstrap. (Chapman and Hall/CRC, 1994). doi:10.1201/9780429246593

    doi:10.1201/9780429246593 1994

  63. [69]

    Dietterich, T. G. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10, 1895–1923 (1998). 39

    1923

  64. [71]

    Es tut mir Leid

    Penedo, G. et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems 37, 30811–30849 (2024). 40 Supplementary information 41 Supplementary Figure 1: Web-based human evaluation interface used for statement-level grading of LLM- generated tax law answers. The interface presents the or...

    2024

  65. [72]

    Classification tasks Assignment of facts, income, or transactions to legally defined tax categories or income types under statutory provisions

  66. [73]

    Fill-in-the-blank tasks Completion of missing legal terms, statutory references, or numerical thresholds in tax- law statements or provisions

  67. [74]

    Sequence tasks Ordering of procedural steps in tax assessments, filings, audits, or administrative processes

  68. [75]

    Text comprehension tasks Interpretation of statutory provisions, administrative guidance, or case law excerpts and explanation of their legal implications

  69. [76]

    Argumentation tasks Justification of tax-law positions through structured legal reasoning and reference to applicable norms

  70. [77]

    Matching tasks Mapping of tax concepts, income types, or legal consequences to predefined categories or classifications

  71. [78]

    Extension tasks Completion of incomplete legal enumerations or factual descriptions based on statutory structure

  72. [79]

    Explanatory tasks Structured explanation of tax-law concepts or mechanisms in coherent natural language

  73. [80]

    Rule extraction tasks Identification of applicable statutory requirements or conditions from legal texts

  74. [81]

    Comparison tasks Systematic comparison of alternative tax treatments, income types, or legal regimes. 50

  75. [82]

    Question generation tasks Formulation of new tax-law questions based on a given statutory provision or legal concept

  76. [83]

    Translation tasks Reformulation of complex tax-law provisions into simplified or non-technical language

  77. [84]

    Comprehension and commentary tasks Analytical discussion of statutory provisions, including contextual interpretation and illustrative examples

  78. [85]

    Accounting and booking tasks Derivation of accounting entries and identification of affected accounts in tax-relevant business transactions

  79. [86]

    Framework analysis tasks Identification of relevant tax-law and legal conditions governing a broader factual constellation

  80. [87]

    Correction tasks Detection and correction of incorrect tax treatments, calculations, or legal assumptions

Showing first 80 references.