REVIEW 4 major objections 5 minor 33 references
Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: An Experimental Study
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that retrieval-augmented generation over official legal texts can match or beat fine-tuning for Austrian VAT case analysis, making expensive fine-tuning unnecessary for useful legal reasoning.
desk verdict A careful applied study of RAG vs. fine-tuning for Austrian VAT reasoning, with real released data, but the headline textbook advantage is in-sample and the real-world comparison is non-significant, so the conclusion that fine-tuning may be unnecessary is not as strong as the abstract suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a retrieval-augmented generation pipeline over the current official legal corpus—the Austrian VAT Act, its Internal Market Regulation annex, and the VAT directive—chunked at 1024 tokens and embedded with text-embedding-ada-002, retrieving the top 5 chunks into a prompt that forces a JSON response with the legal justification before the country. A mocked retrieval variant, which injects the ground-truth relevant legal chunks regardless of retrieval, sets an upper bound on what the LLM can do with perfect retrieval. The prompt design and the separation of retrieval quality from reasoning quality are what make the comparison between RAG and fine-tuning interpretable.
What would settle it
Have two independent VAT expert panels, blind to which system produced each answer, re-judge all 74 textbook answers and all 20 real-world justifications; if their verdicts disagree with the paper's labels on enough cases to reverse the 93.24% versus 89.19% textbook ordering, the claimed advantage of RAG over fine-tuning is not supported. Alternatively, run the same comparison on a fresh set of 100 real client cases; if fine-tuning matches or exceeds RAG, the conclusion that fine-tuning is unnecessary would be falsified.
Extended reading notes
Core claim
The paper's central claim is that retrieval-augmented generation is at least as good as, and in the reported numbers better than, fine-tuning for legal decision-making in Austrian and EU VAT law, and that expensive fine-tuning may not be required for useful reasoning performance. The best RAG configuration identified the correct place of supply in 69 of 74 textbook cases (93.24%), while the fine-tuned GPT-4o model scored 89.19%. On 20 real-world cases from tax consulting practice, RAG gave correct justifications in 16 cases (80%) and the fine-tuned model in 14 (70%); a McNemar test shows this difference is not statistically significant. Combining the two methods performed worse (55%), and even a configuration with perfect retrieval left a small error rate, so the authors frame the systems as decision support rather than full automation.
Load-bearing premise
Every accuracy figure depends on the authors' and their tax experts' manual decisions about what counts as the correct place of supply and the correct legal justification, made without blind review or inter-rater reliability checks.
Editorial extensions
If this is right
- A tax consultancy can deploy a useful VAT assistant by connecting a general-purpose LLM to current legal texts via RAG, avoiding the cost and data requirements of fine-tuning.
- Legal justifications produced by RAG are more often correct than those from a fine-tuned model, so grounding answers in retrievable legal sources is the more reliable route to reducing hallucination.
- Combining RAG with fine-tuning can degrade accuracy, so system builders should evaluate the combination rather than assume it helps.
- The assistant still needs structured client context and access to attachments before it can operate outside a human-in-the-loop support role.
- Because Austrian VAT law is harmonized with EU provisions, the same retrieval-plus-LLM architecture is in principle adaptable to other EU tax jurisdictions.
Reading between the lines
- The paper's own McNemar test means the headline RAG advantage over fine-tuning is not statistically significant at this sample size; the defensible conclusion is that RAG matches fine-tuning at much lower cost, not that it is proven superior.
- Because perfectly retrieved legal context reached 95.95% accuracy with all justifications correct, the largest remaining loss in the real RAG system is retrieval quality; improving chunking and retrieval may yield more gains than further model training.
- A natural extension the authors do not test: measure whether the same RAG configuration transfers to other EU Member States' VAT rules, since the legal sources are harmonized EU directives.
- The poor result of RAG plus fine-tuning suggests the two knowledge sources can conflict; a testable design rule would be to treat fine-tuning and retrieval as alternatives, not complements.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an experimental comparison of retrieval-augmented generation (RAG) and fine-tuning for an LLM-based assistant that determines the place of supply under Austrian and EU VAT law, using 74 textbook cases for iterative development and 20 real-world client inquiries for final evaluation. The headline results are 93.24% accuracy for RAG versus 89.19% for the fine-tuned GPT-4o on the textbook cases, and 80% versus 70% correct justifications on the real-world cases, with a non-significant McNemar test (p = 0.683). Based on these numbers, the paper concludes in Section 10 that expensive fine-tuning may not be required for useful reasoning performance in VAT law, while also acknowledging that current prototypes are not ready for full automation.
Significance. The paper addresses a practically relevant question: whether retrieval augmentation can substitute for costly fine-tuning in a specialized legal domain. It has real strengths: the code, datasets, and results are released in a public repository; the textbook and real-world case distinction is a sensible evaluation design; the Mocked RAG condition in Section 8.1 provides a useful upper bound on retrieval performance; and the authors honestly report the non-significant McNemar test on the real-world cases. If the evaluation were properly out-of-sample, the study would offer a valuable benchmark for tax-consulting practice. However, the central quantitative evidence for the conclusion is undermined by the fact that the headline textbook accuracy was used to select hyperparameters and then reported as the final result, and the only independent comparison is underpowered and non-significant. These issues are fixable within the manuscript's scope, but they currently weaken the paper's main claim.
major comments (4)
- [Sections 6.4, 7.2, and Table 5] The headline RAG-versus-fine-tuning comparison is in-sample. Section 6.4 reports that the RAG configuration (embedding model, chunk size, top k, efSearch) was selected by maximizing accuracy on the 74 textbook VAT cases, and the selected configuration's accuracy of 93.24% is then reported again as the final RAG accuracy in Table 5. Section 7.2 similarly uses accuracy on the same 74 textbook cases to select fine-tuning hyperparameters (Table 4), and the resulting 89.19% in Table 5 is measured on that same set. Since the same 74 cases were used both for configuration selection and final evaluation, the reported 3-case gap (93.24% vs. 89.19%) can reflect selection on noise rather than a true difference. The paper should either re-run the comparison on a genuinely held-out test split (e.g., nested cross-validation or a development/test split of the 74 cases) or explicitly restrict the conclusion to in-sample performance and provide appropriate caveats.
- [Section 8.2 and Section 10] The only out-of-sample comparison is the 20 real-world cases, where RAG achieves 80% and the fine-tuned model achieves 70%, with McNemar's test giving p = 0.683. This is non-significant, and the paper says so in Section 8.2. Nevertheless, Section 10 concludes that 'the RAG system consistently outperformed the fine-tuned LLM' and uses this to suggest that fine-tuning may not be required. Non-significance at n = 20 does not establish superiority, and it certainly does not support a general claim that fine-tuning is unnecessary. The conclusion should be tempered to say that the data do not show a statistically significant advantage for either method, and that the only apparent advantage (on the textbook cases) is based on an in-sample evaluation.
- [Sections 4.1, 8.1, and 8.2] The ground truth for the textbook cases and the manual correctness judgments for justifications were constructed by the authors and tax experts without a documented blind evaluation or inter-rater reliability check. All reported accuracy figures depend on this reference standard, and the subjective judgment of what counts as a legally correct answer or a sound justification is load-bearing. If the reference standard is biased or inconsistent, every comparative claim in the paper is affected. The authors should report the number and qualifications of the evaluators, the exact evaluation protocol, and an agreement measure such as Cohen's kappa, and they should ideally use blinded evaluation with pre-specified answer keys.
- [Section 4.2 and Section 9.2] The real-world cases were enriched by extracting details from the tax advisors' responses and adding them to the client inquiries. The paper asserts that this enrichment was done 'without revealing the correct answer,' but the advisors' responses contain legal reasoning and conclusions; if any conclusion-bearing content leaked into the enriched inquiry, the evaluation becomes partially circular. The authors need to make the extraction procedure transparent: what was included, what was excluded, and why the extracted information could not have disclosed the correct legal outcome. Examples of original versus enriched case descriptions would help the reader assess this risk.
minor comments (5)
- [Section 4.1] The sentence 'which serve to evaluate the the RAG system's retrieval performance' contains a duplicated 'the'.
- [Section 2.1] In the sentence beginning 'However, according to Sun [14], the integration of LLMs into legal practice pos challenges', 'pos' should be 'poses'.
- [Table 5] Reporting the accuracy as percentages alone obscures how small the differences are on 74 cases; adding case counts (e.g., 69/74, 66/74) and confidence intervals would make the uncertainty visible.
- [Section 5.1] The prompt development process is described qualitatively; stating the number of prompt variants tried and the criterion for selecting the final prompt would improve reproducibility.
- [Section 8.1] The discussion of the manual justification evaluation would benefit from a brief description of how disagreements among evaluators were resolved, if there were any.
Circularity Check
The headline RAG-over-fine-tuning comparison rests on accuracy numbers computed on the same 74 textbook cases used to select the prompts, RAG hyperparameters, and fine-tuning hyperparameters, so the reported gap is partly an in-sample selection artifact.
-
fitted input called prediction
[Section 6.4 and Table 5 (Section 8.1)]
"We experimented with various RAG configurations and found that, for the task of automated identification of the place of supply in the textbook V AT cases, the best accuracy (93.24 %) was achieved using the text-embedding-ada-002 model, a chunk size of 1024 tokens, a chunk overlap of 0, top k set to 5, and efSearch set to 800."
The 93.24% is by construction the maximum accuracy obtained by searching over multiple RAG configurations on the same 74 textbook cases that Table 5 then reports as the RAG system's 'final evaluation' accuracy. The configuration was selected to maximize this exact number, so the headline RAG accuracy is an in-sample optimum rather than an independent prediction. With only 74 cases, the gap between 93.24% and 89.19% is three cases, and selection over the many configurations in Tables 1-3 can produce such a gap by chance. The conclusion that 'expensive fine-tuning may not be required' therefore leans on a number that is partly manufactured by the selection procedure.
-
fitted input called prediction
[Section 5.1 and Table 5]
"To determine the best prompt formulation, we used the textbook V AT cases on the question of the place of supply. We evaluated the quality of the prompts based on the accuracy of the identified places of supply..."
The final prompt was selected by measuring accuracy on the very same 74 textbook cases that are later used as the final evaluation set in Table 5. Every accuracy in Table 5 is produced with this prompt, so the reported numbers are the criterion values used to choose the prompt, not an out-of-sample estimate. This contaminates all configurations in the textbook comparison, including the Mocked RAG upper bound, and makes the paper's central textbook-case evidence a fitted value rather than a fresh prediction.
1 more flagged steps
-
fitted input called prediction
[Section 7.2, Tables 4 and 5]
"Table 4 shows the accuracy in determining the place of supply for the various fine-tuned models... Using a batch size of 16, a learning rate multiplier of 2.8, and three training epochs—settings, that yielded the best results during the fine-tuning process with 85.14 % accuracy... we fine-tuned GPT-4o on all 758 examples... The resulting model was then used in the final evaluation."
Fine-tuning hyperparameters were selected on the basis of accuracy on the same textbook VAT cases that appear as 'Fine-tuned LLM 89.19%' in Table 5. The final reported fine-tuned accuracy is therefore the result of choosing among nine hyperparameter settings on the evaluation set, not an independent measurement of the chosen model's performance. The paper does include an out-of-sample real-world comparison, but it reports McNemar p = 0.683, so the fine-tuning-versus-RAG conclusion relies mainly on the in-sample textbook gap that was used as the selection criterion.
full rationale
The paper is an empirical engineering study, not a formal derivation, so the circularity here is not definitional but statistical: the central textbook-case accuracy comparison is in-sample. The RAG configuration, the prompt, and the fine-tuning hyperparameters were all selected by optimizing accuracy on the same 74 textbook cases that are then reported as the final evaluation in Table 5. In particular, the RAG row's 93.24% is exactly the best value found during configuration search, so reporting it as evidence that RAG outperforms fine-tuning is reporting the optimization objective as the outcome. The 20 real-world cases provide a more independent check, but the paper itself discloses that the inputs were enriched with details extracted from the advisors' explanations, which are also the basis of the ground-truth evaluation, creating a mild information leak; moreover, the real-world RAG-versus-fine-tuning difference is not statistically significant. There is no load-bearing self-citation chain or uniqueness-theorem argument. Because the headline comparison is substantially an in-sample selection artifact rather than a genuinely out-of-sample prediction, a score of 6 is appropriate: partial circularity via fitted inputs being reported as predictions.
Assumptions & free parameters
free parameters (8)
- RAG chunk size =
1024 tokens
- RAG top k =
5
- efSearch =
800
- Chunk overlap =
0
- Embedding model =
text-embedding-ada-002
- Fine-tuning batch size =
16
- Fine-tuning learning rate multiplier =
2.8 (text) / 2.5 (Table 4)
- Fine-tuning epochs =
3
assumptions (6)
- domain assumption The selected Austrian legal documents (VAT Act, Annex/Internal Market Regulation, and Ministry Directive) are complete, current as of December 1, 2024, and correctly represent the law used for ground truth.
- domain assumption The manually determined place of supply for the 74 textbook cases is correct ground truth.
- domain assumption Tax experts' manual judgments of justification correctness are valid and unbiased measures of legal reasoning quality.
- ad hoc to paper Enriching the real-world case descriptions with details extracted from the advisors' responses does not reveal the correct answer.
- domain assumption The 758-example fine-tuning dataset is representative of the VAT cases the models will encounter.
- standard math McNemar's test with Yates' correction is appropriate for comparing paired accuracies on 20 real-world cases.
Cite this review
Pith. "Pith review of Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: An Experimental Study." pith.science (2026). https://pith.science/paper/OF5NYUWJ
@misc{pith2026250708468,
author = {Pith},
title = {Pith review of: Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: An Experimental Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/OF5NYUWJ}},
note = {Machine review of arXiv:2507.08468}
}
read the original abstract
This paper provides an experimental evaluation of the capability of large language models (LLMs) to assist in legal decision-making within the framework of Austrian and European Union value-added tax (VAT) law. In tax consulting practice, clients often describe cases in natural language, making LLMs a prime candidate for supporting automated decision-making and reducing the workload of tax professionals. Given the requirement for legally grounded and well-justified analyses, the propensity of LLMs to hallucinate presents a considerable challenge. The experiments focus on two common methods for enhancing LLM performance: fine-tuning and retrieval-augmented generation (RAG). In this study, these methods are applied on both textbook cases and real-world cases from a tax consulting firm to systematically determine the best configurations of LLM-based systems and assess the legal-reasoning capabilities of LLMs. The findings highlight the potential of using LLMs to support tax consultants by automating routine tasks and providing initial analyses, although current prototypes are not ready for full automation due to the sensitivity of the legal domain. The findings indicate that LLMs, when properly configured, can effectively support tax professionals in VAT tasks and provide legally grounded justifications for decisions. However, limitations remain regarding the handling of implicit client knowledge and context-specific documentation, underscoring the need for future integration of structured background information.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
M. Sako, How generative ai fits into knowledge work, Communications of the ACM 67 (4) (2024) 20–22, https://doi.org/10.1145/3638567
- [2]
- [3]
-
[4]
European Comission, Value-added tax, https://commission.europa.eu/ strategy-and-policy/eu-budget/long-term-eu-budget/2021-2027/ revenue/own-resources/value-added-tax_en , accessed: 2025-02-15
work page 2021
-
[5]
M. Luketina, A. Benkel, C. G. Schuetz, Using large language models for legal decision-making in Austrian value-added tax law: An experimental study (supplementary material) (2025). URL https://doi.org/10.6084/m9.figshare.28943927
-
[6]
R. Hamdani, T. Bonald, F. Malliaros, N. Holzenberger, F. Suchanek, The factuality of large language models in the legal domain, Proceedings of the 33rd ACM International Conference on Information and Knowledge Man- agement (2024) 3741–3746https://doi.org/10.1145/3627673.3679961
arXiv 2024
-
[7]
J. J. Nay, D. Karamardian, S. B. Lawsky, W. Tao, M. Bhat, R. Jain, A. T. Lee, J. H. Choi, J. Kasai, Large language models as tax attorneys: a case study in legal capabilities emergence, Philosophical Transactions of the Royal Society 382 (2270) (2024) 20230159, https://doi.org/10.48550/ arXiv.2306.07075
work page Pith review arXiv doi:10.48550/arxiv.2306.07075 2024
-
[8]
Large Language Models are legal but they are not: Making the case for a powerful LegalLLM
T. Jayakumar, F. Farooqui, L. Farooqui, Large Language Models are legal but they are not: Making the case for a powerful Legal LLM, arXiv preprint arXiv:2311.08890 (2023). URL https://doi.org/10.48550/arXiv.2311.08890
work page Pith review arXiv doi:10.48550/arxiv.2311.08890 2023
Show all 33 references
-
[9]
Z. Fei, X. Shen, D. Zhu, F. Zhou, Z. Han, S. Zhang, K. Chen, Z. Shen, J. Ge, LawBench: Benchmarking legal knowledge of large language models, arXiv preprint arXiv:2309.16289 (2023). URL http://arxiv.org/abs/2309.16289
2023 arXiv
-
[10]
Savelka, K
J. Savelka, K. D. Ashley, The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts, Frontiers in Artificial Intelligence 6 (2023) 1279794
2023
-
[11]
J. Savelka, Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts, in: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, 2023, pp. 447–451. 24
2023
-
[12]
Ammar, A
A. Ammar, A. Koubaa, B. Benjdira, O. Nacar, S. Sibaee, Prediction of arabic legal rulings using large language models, Electronics 13 (4) (2024) 764
2024
-
[13]
G´ orski, B
L. G´ orski, B. Ku´ zniacki, M. Almada, K. Tyli´ nski, M. Calvo, P. M. Asnaghi, L. Almada, H. I˜ niguez, F. Rubianes, O. Pera, et al., Exploring explainable ai in the tax domain, Artificial intelligence and law (2024) 1–29
2024
- [14]
- [15]
-
[16]
L. Xu, C. Hu, H. Zhang, J. Zhai, W. Tang, Y. Li, Z. Peng, Q. Chen, S. Sun, A. Ji, et al., Surpassing Human Counterparts: A Breakthrough Achievement of Large Language Models in Professional Tax Qualification Examinations in China, in: 2024 IEEE Conference on Artificial Intellig...
2024
-
[17]
Wiratunga, R
N. Wiratunga, R. Abeyratne, L. Jayawardena, K. Martin, S. Massie, I. Nkisi-Orji, R. Weerasinghe, A. Liret, B. Fleisch, CBR-RAG: case-based reasoning for retrieval augmented generation in LLMs for legal question an- swering, in: International Conference on Case-Based Reasoning,...
2024
-
[18]
at/en/statistics/national-economy-and-public-finance/ public-finance/public-finance/tax-revenue , accessed: 2024-10- 19
Statistics Austria, Tax revenue, https://www.statistik. at/en/statistics/national-economy-and-public-finance/ public-finance/public-finance/tax-revenue , accessed: 2024-10- 19
2024
-
[19]
wxe?Abfrage=Bundesnormen&Gesetzesnummer=10004873, accessed: 2024- 10-01
Austrian V AT Act, https://www.ris.bka.gv.at/GeltendeFassung. wxe?Abfrage=Bundesnormen&Gesetzesnummer=10004873, accessed: 2024- 10-01
2024
-
[20]
European Union, Council Directive 2006/112/EC of 28 November 2006 on the common system of value added tax, https://eur-lex.europa.eu/ eli/dir/2006/112/oj/eng, accessed: 2025-02-15
2006
-
[21]
bmf.gv.at/findok/volltext(suche:Standardsuche)?dokumentId= 6fd85cac-e581-4d6e-953d-a73e25d2bda0 , accessed: 2024-11-02
Austrian Federal Ministry of Finance, Ustr 2000, https://findok. bmf.gv.at/findok/volltext(suche:Standardsuche)?dokumentId= 6fd85cac-e581-4d6e-953d-a73e25d2bda0 , accessed: 2024-11-02
2000
-
[22]
Wieringa, Design science methodology for information systems and soft- ware engineering, Springer, 2014
R. Wieringa, Design science methodology for information systems and soft- ware engineering, Springer, 2014. 25
2014
- [23]
-
[24]
Berger, M
W. Berger, M. Wakounig, Umsatzsteuer kompakt 2022/2023, 9th Edition, Linde, 2023
2022
-
[25]
OpenAI, Prompt Engineering, https://platform.openai.com, accessed: 2025-12-18
2025
- [26]
-
[27]
PyMuPDF, Using PyMuPDF in an RAG Chatbot Environment, https: //github.com/pymupdf/RAG, accessed: 2025-05-31
2025
-
[28]
azure.cn/en-us/search/vector-search-how-to-chunk-documents , accessed: 2025-03-15
Azure AI Search, Chunk documents in vector search, https://docs. azure.cn/en-us/search/vector-search-how-to-chunk-documents , accessed: 2025-03-15
2025
-
[29]
microsoft.com/en-us/javascript/api/@azure/search-documents/ hnswparameters?view=azure-node-latest, accessed: 2025-02-14
Azure AI Search, HnswParameters interface, https://learn. microsoft.com/en-us/javascript/api/@azure/search-documents/ hnswparameters?view=azure-node-latest, accessed: 2025-02-14
2025
-
[30]
Azure OpenAI Services, Customize a model with fine-tuning, https://learn.microsoft.com/en-us/azure/ai-services/openai/ how-to/fine-tuning, accessed: 2024-11-19
2024
-
[31]
Kollmann, Skriptum Umsatzsteuer (152) – Band 1, Akademie der Steuerberater:innen und Wirtschaftspr¨ ufer:innen (2024)
G. Kollmann, Skriptum Umsatzsteuer (152) – Band 1, Akademie der Steuerberater:innen und Wirtschaftspr¨ ufer:innen (2024)
2024
-
[32]
B¨ urgler, Skriptum Umsatzsteuer (152) – Band 2: Beispielband, Akademie der Steuerberater:innen und Wirtschaftspr¨ ufer:innen (2024)
C. B¨ urgler, Skriptum Umsatzsteuer (152) – Band 2: Beispielband, Akademie der Steuerberater:innen und Wirtschaftspr¨ ufer:innen (2024)
2024
-
[33]
OpenAI, Chat Completions, https://platform.openai.com/docs/ api-reference/chat, accessed: 2025-05-30. 26
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.