Pith. sign in

REVIEW 4 major objections 6 minor 21 references

BR-TaxQA-R: A Dataset for Question Answering with References for Brazilian Personal Income Tax Law, including case law

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new Brazilian tax-law QA dataset with references supports a retrieval-augmented pipeline that beats commercial chatbots on response relevancy while trading off factual correctness and fluency.

desk verdict New Portuguese legal QA dataset is a real resource; the RAGAS-based superiority claim is confounded by shared evaluator and system models. read the letter →

arxiv 2505.15916 v1 pith:W4UMLZS3 submitted 2025-05-21 cs.CL cs.AI

Juvenal Domingos Júnior , Augusto Faria , E. Seiti de Oliveira , Erick de Brito , Matheus Teotonio , Andre Assumpção , Diedre Carmo , Roberto Lotufo
show 1 more author
Jayr Pereira
This is my paper
classification cs.CLcs.AI
keywords Brazilianpersonalincometaxquestionansweringwithreferencesretrieval-augmentedgenerationlegalNLPcaselawRAGevaluationdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces BR-TaxQA-R, a dataset of 715 questions and answers drawn from Brazil's official 2024 personal income tax FAQ, paired with the legal statutes those answers cite and 7,204 administrative rulings from the country's tax appeals council. Using this dataset, the authors build a retrieval-augmented question-answering pipeline and compare several document-segmentation strategies against commercial web-search chatbots. The paper's central claim is that a simple sliding-window segmentation enriched with case law scores highest on response relevancy (0.829), beating all commercial tools, while the commercial tools lead on factual correctness and fluency. The authors argue this is a real trade-off between legal traceability and linguistic polish, and that automatic metrics alone cannot establish legal validity, so human expert review remains necessary for high-stakes tax guidance.

What carries the argument

The central object is BR-TaxQA-R, a dataset that pairs 715 official questions and answers on Brazilian personal income tax with 478 source legal documents and 7,204 administrative rulings. The argument is carried by a retrieval-augmented generation pipeline: legal texts are segmented either into overlapping 2048-token windows with 1024-token stride or by recursive splitting on legal-hierarchy boundaries, embedded as dense vectors, searched by nearest-neighbor over a vector index, and passed to a language model prompted to answer from retrieved context only and to list cited legal sources at the end. The evaluation machinery is an automatic QA metric suite that derives response relevancy from the cosine similarity between the original question and proxy questions generated from the candidate answer, and that scores factual correctness, semantic similarity, BLEU, and ROUGE-L against the ground-truth answer.

What would settle it

Have licensed tax lawyers blind-score a sample of the generated answers on legal validity, source traceability, and completeness; if their rankings do not reproduce the automatic ordering, for instance by judging the commercial chatbots' answers as legally sound as often as the RAG system's, the claimed trade-off is unsupported.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a domain-specific retrieval pipeline grounded in official tax sources can match user query intent better than general-purpose commercial assistants: the sliding-window configuration with administrative rulings reaches a response-relevancy score of 0.829, ahead of the strongest commercial chatbot at 0.738. The same configuration reaches 0.327 factual correctness, below commercial tools' 0.389-0.469 range, and its BLEU and ROUGE-L scores trail or tie the best commercial output. The paper reads these numbers as evidence that adding statutory text and case law to a retrieval context improves relevance and legal grounding at the cost of linguistic fluency, and that high fluency scores do not guarantee that an answer is legally sound. It therefore concludes that the dataset should be used for full-pipeline evaluation and that human expert judgment is indispensable before any such answer is relied upon.

Load-bearing premise

The whole comparison depends on treating automatically computed relevancy and correctness scores as valid measures of answer quality for legal questions, yet the paper itself warns that such metrics can miss legal nuance.

Editorial extensions

If this is right

  • Adding administrative rulings to the retrieval corpus raises response relevancy for the custom RAG system, so case law is a usable lever for aligning answers with user intent.
  • Optimizing for fluency-oriented metrics such as BLEU and ROUGE will not by itself produce legally trustworthy answers; the paper's evidence ties high fluency to unsupported or ungrounded content in commercial outputs.
  • The sliding-window segmentation with case law configuration provides a strong reference baseline for future full-pipeline evaluations on this dataset.
  • The 715 questions, 478 source documents, and 7,204 rulings form a reusable testbed for probing legal retrieval outside the 715 annotated questions.
  • A system that scores high on factual correctness in this benchmark should not be treated as legally reliable, because the paper reports fluent but ungrounded answers from the commercial tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural follow-up is a human-annotation study in which tax lawyers score the same answers on legal validity; if expert rankings diverge from the response-relevancy ordering, the paper's trade-off story would need revision.
  • The pipeline can be transferred to other administrative domains with official Q&A booklets, such as social security or customs law, where statutes and administrative rulings are similarly structured.
  • Because the dataset contains only one year of rulings, a testable extension is whether the relevancy gain comes from recency of rulings or simply from having extra legal text in the retrieval pool.
  • Metric stability could be audited by recomputing the evaluations with a different judge model; the paper's conclusions would be stronger if the rankings survive that substitution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BR-TaxQA-R, a dataset for question answering with references in the domain of Brazilian personal income tax law. The dataset contains 715 questions from the official 2024 Receita Federal Q&A document, enriched with statutory sources explicitly and implicitly cited in the answers, plus a set of 7,204 CARF administrative rulings from 2023 used as case law. The authors implement a custom RAG pipeline using OpenAI embeddings (text-embedding-3-small) and GPT-4o-mini for generation, compare two segmentation strategies (sliding-window and recursive splitting) with and without case-law supplementation, and benchmark the resulting systems against ChatGPT, Perplexity.ai, and Grok using RAGAS metrics (Response Relevancy, Factual Correctness, Semantic Similarity, BLEU, ROUGE-L). The central empirical claim is that the sliding-window segmentation with case law achieves the highest Response Relevancy (0.829), outperforming all commercial tools, while commercial tools score higher on Factual Correctness and fluency, suggesting a trade-off between legal grounding and linguistic fluency. The paper also argues that human expert evaluation remains essential for legal QA.

Significance. If the dataset and results hold, BR-TaxQA-R is a valuable contribution to legal NLP in Brazilian Portuguese: it is the first public dataset combining official RFB Q&A with both statutory references and CARF case law, and it includes explicit, implicit, and formatted reference fields that enable end-to-end RAG evaluation. The public Hugging Face release is a concrete resource for future work. The paper is also honest in acknowledging that automated metrics are insufficient for legal validity. However, the headline empirical claim about outperforming commercial tools is load-bearing and is currently weakened by evaluation-methodology issues that need to be addressed before the trade-off conclusion can be accepted.

major comments (4)
  1. [§4.3 and §5, Table 4] The headline claim that 'sliding-window segmentation with case law' achieves the highest Response Relevancy (0.829) is not yet established, because the RAGAS Response Relevancy metric is computed with gpt-4o-mini for question generation and text-embedding-3-small for embeddings, the same model family used to build the custom RAG system. This shared model and embedding space can inflate the metric for the custom RAG's answers relative to answers from ChatGPT, Perplexity, and Grok. The paper reports no control for this confound, such as an independent evaluator model, a different embedding model, or human relevance judgments. Please add such a control or temper the outperformance claim accordingly.
  2. [§5 and §6] The paper itself concedes in Section 6 that high-scoring RAG answers lacked critical legal nuance and that metrics such as semantic similarity or BLEU/ROUGE alone are insufficient to guarantee legal validity. Since the trade-off conclusion is derived from these same automated metrics, the concession directly undermines the strength of that conclusion. A human expert evaluation of a sample of answers from both the custom RAG and the commercial tools is needed to validate that the direction of the reported trade-off is real rather than an artifact of the chosen metrics.
  3. [§5, Table 4] All reported metrics appear to come from a single evaluation run, with no confidence intervals, standard deviations, or significance tests. Differences between configurations are small (for example, Response Relevancy 0.829 with sliding-window plus case law versus 0.811 with recursive segmentation plus case law; BLEU 0.190 versus 0.178), so without repeated runs or statistical testing the ranking of segmentation strategies and the claim that case law improves relevance are not robust.
  4. [§2.3] The case-law collection is restricted to 2023 CARF rulings, selected via keyword matching against question text and a temporal filter, but the paper does not report how many rulings were retrieved, how many were excluded by the filter, or any validation that the selected rulings are actually relevant to the 2024 questions. This makes the 'case law' component difficult to reproduce and weakens the interpretation that incorporating jurisprudence improves Response Relevancy. Please provide these statistics and at least a manual sample check of relevance.
minor comments (6)
  1. [§2.1] The field name 'all_formated_references' contains a typo; it should be 'all_formatted_references'. The same typo appears in the description of the questions set.
  2. [Table 2] Table 2 is difficult to read: the row labeled '≥104' appears to be a typo for '≥10', and the bottom row 'question max' is ambiguous. Please reformat the table for clarity and correct the typo.
  3. [Abstract and §1] The abstract states that the custom RAG 'outperforms commercial systems in Response Relevancy', while the Introduction says 'closed-source commercial tools employing LLM-based search pipelines achieve superior performance'. These are consistent, but the phrasing should be tightened to make explicit that the custom system is superior only on Response Relevancy and not on overall performance.
  4. [§4.1] The prompt design asks the model to include citations of applicable legal sources at the end of the response and to avoid mid-response citations, but the evaluation section does not specify whether the structured citation list is part of the candidate string that is scored by the automatic metrics. Please clarify how citations are handled in the metric computation.
  5. [§4.3] In the description of ROUGE, the phrase 'based on sensitivity (which is the fraction of correctly selected data from all relevant entailments)' is confusing and does not match standard ROUGE terminology; consider replacing 'sensitivity' with the standard recall/precision/F1 formulation.
  6. [§2.3] The analogy to retrospective studies indexed by 'date of death' is out of place and does not clarify the temporal-filter decision; consider removing or elaborating the connection.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset and benchmark are grounded in official RFB/CARF documents, and the headline claims are empirical comparisons, not derivations from the paper's own definitions or fitted parameters.

full rationale

BR-TaxQA-R is constructed from the official RFB 2024 Q&A document and CARF administrative rulings, so the ground truth is external to the authors' system. The custom RAG system is a standard retrieval-generation pipeline (text-embedding-3-small indexing plus gpt-4o-mini generation), and Table 4 compares it against commercial tools on RAGAS metrics. No parameter is fitted to the evaluation metric, no result is obtained by renaming an input, and the central response-relevancy score is not defined in terms of the RAG system's own retrieval score. The only direct self-citation, [13], is used to position the dataset as an extension of prior work; it does not carry the empirical claims. The shared use of OpenAI models in both the custom RAG and the RAGAS evaluator is a legitimate external-validity concern about evaluation fairness, and the paper itself (Section 6) concedes that automated metrics cannot guarantee legal validity. Those are reliability caveats, not circular reductions: the response-relevancy advantage would need independent human or cross-model validation, but the paper does not claim to derive that advantage from its own definitions. Under the rule that only definitional or fitted-input reductions count as circularity, this paper is not circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central evaluation runs on three externally supplied pillars: the RFB document as authoritative ground truth, CARF ruling selection criteria chosen by the authors, and RAGAS metrics using OpenAI models; each adds cost the paper does not independently justify, and the paper's own conclusion calls for human expert evaluation.

free parameters (4)
  • top_k retrieval depth = not reported
    Number of retrieved segments fed to GPT-4o-mini is not stated in the paper, but it directly controls the context and therefore the answers.
  • sliding window size / stride = 2048 tokens / 1024 tokens
    Chosen in Section 4.1 without ablation; affects the segmentation and retrieval quality.
  • recursive splitter chunk size / overlap = 1000 chars / 100 chars
    Chosen in Section 4.1; affects segmentation and retrieval quality.
  • LLM generation temperature / max tokens = not reported
    Answer generation settings for gpt-4o-mini are not reported, affecting fluency and stochastically varying scores.
assumptions (4)
  • domain assumption The official RFB 2024 Q&A document is authoritative ground truth for the answers.
    Invoked throughout Section 2.1 and Section 3.1; all 715 questions and answers are extracted from this single document with no independent legal validation.
  • domain assumption RAGAS metrics computed with GPT-4o-mini and text-embedding-3-small reliably measure answer quality and legal adequacy.
    Invoked in Section 4.3 and used for all Table 4 scores; the authors themselves state in Section 6 that these metrics are insufficient to guarantee legal validity.
  • ad hoc to paper CARF rulings published in 2023 are relevant and current for interpreting 2024 Q&A topics.
    Introduced in Section 2.3 as a temporal filter; only 2023 rulings were scraped, excluding earlier years without justification.
  • ad hoc to paper LLM-assisted deduplication of legal references, followed by manual verification, yields an accurate reference set.
    Used in Section 2.1 to merge duplicate document references; no quantitative accuracy check is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BR-TaxQA-R: A Dataset for Question Answering with References for Brazilian Personal Income Tax Law, including case law." pith.science (2026). https://pith.science/paper/W4UMLZS3

@misc{pith2026250515916,
  author       = {Pith},
  title        = {Pith review of: BR-TaxQA-R: A Dataset for Question Answering with References for Brazilian Personal Income Tax Law, including case law},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W4UMLZS3}},
  note         = {Machine review of arXiv:2505.15916}
}
read the original abstract

This paper presents BR-TaxQA-R, a novel dataset designed to support question answering with references in the context of Brazilian personal income tax law. The dataset contains 715 questions from the 2024 official Q\&A document published by Brazil's Internal Revenue Service, enriched with statutory norms and administrative rulings from the Conselho Administrativo de Recursos Fiscais (CARF). We implement a Retrieval-Augmented Generation (RAG) pipeline using OpenAI embeddings for searching and GPT-4o-mini for answer generation. We compare different text segmentation strategies and benchmark our system against commercial tools such as ChatGPT and Perplexity.ai using RAGAS-based metrics. Results show that our custom RAG pipeline outperforms commercial systems in Response Relevancy, indicating stronger alignment with user queries, while commercial models achieve higher scores in Factual Correctness and fluency. These findings highlight a trade-off between legally grounded generation and linguistic fluency. Crucially, we argue that human expert evaluation remains essential to ensure the legal validity of AI-generated answers in high-stakes domains such as taxation. BR-TaxQA-R is publicly available at https://huggingface.co/datasets/unicamp-dl/BR-TaxQA-R.

Figures

Figures reproduced from arXiv: 2505.15916 by the authors.

Figure 1
Figure 1. Illustration of the trade-off between contextual precision and linguistic fluency [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

  1. [1]

    Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library (2025),https://arxiv.org/abs/ 2401.08281

    arXiv 2025

  2. [2]

    In: Aletras, N., De Clercq, O

    Es, S., James, J., Espinosa Anke, L., Schockaert, S.: RAGAs: Automated eval- uation of retrieval augmented generation. In: Aletras, N., De Clercq, O. (eds.) Proceedings of the 18th Conference of the European Chapter of the Associa- tion for Computational Linguistics: System Demonstrations. pp. 150–158. As- sociation for Computational Linguistics, St. Juli...

    work page 2024

  3. [3]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Feng, Y., Li, C., Ng, V.: Legal case retrieval: A survey of the state of the art. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 6472–6485 (2024)

    work page 2024

  4. [4]

    In: EMNLP (1)

    Karpukhin, V., Oguz, B., Min, S., Lewis, P.S., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. In: EMNLP (1). pp. 6769–6781 (2020)

    work page 2020

  5. [5]

    AI Open (2024)

    Lai, J., Gan, W., Wu, J., Qi, Z., Yu, P.S.: Large language models in law: A survey. AI Open (2024)

    2024

  6. [6]

    In: Proceedings of the 34th Interna- tional Conference on Neural Information Processing Systems

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: Proceedings of the 34th Interna- tional Conference on Neural Information Processing Systems. NIPS ’20, Curran Associates Inc., Red Hook, NY...

    2020

  7. [7]

    In: Text Summarization Branches Out

    Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. Association for Computational Linguis- tics, Barcelona, Spain (Jul 2004),https://aclanthology.org/W04-1013/

    2004

  8. [8]

    URL https://arxiv

    Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C.D., Ho, D.E.: Hallucination-free? assessing the reliability of leading ai legal research tools. URL https://arxiv. org/abs/2405.20362 (2024)

    arXiv 2024

Show all 21 references
  1. [9]

    Associação Brasileira de Jurimetria (2022),https://livro.abj.org.br/, acesso em: 6 maio 2025

    Okamoto, R.F., Trecenti, J.: Metodologia de pesquisa jurimétrica. Associação Brasileira de Jurimetria (2022),https://livro.abj.org.br/, acesso em: 6 maio 2025

    2022

  2. [10]

    In: CEUR Workshop Proceedings

    Oro, E., Granata, F.M., Lanza, A., Bachir, A., De Grandis, L., Ruffolo, M.: Eval- uating retrieval-augmented generation for question answering with large language models. In: CEUR Workshop Proceedings. vol. 3762, pp. 129–134 (2024) BR-TaxQA-R 15

    2024

  3. [11]

    In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)

    2002

  4. [12]

    In: Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval

    Paul, S., Bhatt, R., Goyal, P., Ghosh, S.: Legal statute identification: A case study using state-of-the-art datasets and methods. In: Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2231–2240 (2024)

    2024

  5. [13]

    In: Brazilian Conference on Intelligent Systems

    Presa, J.P.C., Camilo Junior, C.G., Oliveira, S.S.T.d.: Evaluating large language models for tax law reasoning. In: Brazilian Conference on Intelligent Systems. pp. 460–474. Springer (2024)

    2024

  6. [14]

    Receita Federal do Brasil: Perguntas e Respostas IRPF 2024.https: //www.gov.br/receitafederal/pt-br/centrais-de-conteudo/publicacoes/ perguntas-e-respostas/dirpf/pr-irpf-2024.pdf/view(2024), accessed: 2025-05-10

    2024

  7. [15]

    arXiv preprint arXiv:2407.12873 (2024)

    Roychowdhury, S., Soman, S., Ranjani, H., Gunda, N., Chhabra, V., Bala, S.K.: Evaluation of rag metrics for question answering in the telecom domain. arXiv preprint arXiv:2407.12873 (2024)

    2024 arXiv

  8. [16]

    Journal of Information and Data Management15(1), 206–215 (2024)

    da Silva Junior, D., dos Santos Corval, P.R., de Oliveira, D., Paes, A.: Datasets for portuguese legal semantic textual similarity. Journal of Information and Data Management15(1), 206–215 (2024)

    2024

  9. [17]

    Language Resources and Evaluation pp

    Siqueira, F.A., Vitório, D., Souza, E., Santos, J.A., Albuquerque, H.O., Dias, M.S., Silva, N.F., de Carvalho, A.C., Oliveira, A.L., Bastos-Filho, C.: Ulysses tesemõ: a new large corpus for brazilian legal and governmental domain. Language Resources and Evaluation pp. 1–20 (2024)

    2024

  10. [18]

    arXiv preprint arXiv:2406.15313 (2024)

    Su, W., Hu, Y., Xie, A., Ai, Q., Que, Z., Zheng, N., Liu, Y., Shen, W., Liu, Y.: Stard: A chinese statute retrieval dataset with real queries issued by non- professionals. arXiv preprint arXiv:2406.15313 (2024)

    2024 arXiv

  11. [19]

    Language Resources and Evaluation pp

    Vitório, D., Souza, E., Martins, L., da Silva, N.F., de Carvalho, A.C.P.d.L., Oliveira, A.L., de Andrade, F.E.: Building a relevance feedback corpus for legal information retrieval in the real-case scenario of the brazilian chamber of deputies. Language Resources and Evaluatio...

    2024

  12. [20]

    intelligent justice

    Wang, N., Tian, M.Y.: “intelligent justice”: human-centered considerations in china’s legal ai transformation. AI and Ethics3(2), 349–354 (2023)

    2023

  13. [21]

    In: Representation Learning for Natural Language Processing, pp

    Xiao, C., Liu, Z., Lin, Y., Sun, M.: Legal knowledge representation learning. In: Representation Learning for Natural Language Processing, pp. 401–432. Springer Nature Singapore Singapore (2023)

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.