REVIEW 5 major objections 5 minor 37 references
LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval
T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A relevance-check and query-refinement step improves bilingual legal question answering.
desk verdict A modest but legitimate empirical contribution: a relevance-check/query-refinement RAG variant evaluated on a new Bangla legal QA set, with real soft spots around OCR fidelity and overclaimed baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the added relevance-check and query-refinement stage, implemented as a separate small language model placed between the retriever and the final answer generator. After the retriever returns top chunks for a user query, this model decides whether each chunk is actually relevant; if it decides they are not, it rewrites the query while preserving the user's intent, and the system retrieves again. The loop repeats at most three times, and only chunks that pass the check reach the generator, so irrelevant retrieved material is filtered out before answer writing. This stage is what distinguishes the advanced pipeline from the vanilla one and is credited with the measured improvements.
What would settle it
Run the same question set against manually corrected transcripts of a sample of gazette pages and compare retrieval and answer accuracy with the OCR-derived versions; if the corrected transcripts give substantially better scores, then at least part of the claimed gain rests on OCR quality rather than the RAG design itself.
Extended reading notes
Core claim
The paper's central empirical claim is that inserting a relevance-check and query-refinement component into a retrieval-augmented question-answering pipeline improves retrieval and answer quality on a low-resource legal corpus. Working from the Bangladesh Police Gazettes, the authors preprocess 13 mixed English–Bangla gazette documents, build a vector index of chunks, and compare a standard pipeline that sends retrieved chunks straight to an answer generator with an advanced pipeline that first asks a separate language model whether the chunks are relevant, reformulating the query up to three times when they are not. On a curated 168-pair Bangla test set spanning factual, temporal, statistical, dialectal, spelling-error, and out-of-context questions, the advanced pipeline attains higher human evaluation scores and higher mean cosine similarity with lower standard deviation across all three answer generators; the best configuration reaches 3.70 out of 5 on human evaluation and 0.82 mean cosine similarity.
Load-bearing premise
The pipeline depends on the OCR step transcribing mixed Bangla–English gazette pages faithfully enough to preserve legal terms, page references, and numbers, yet the paper reports no measurement of OCR accuracy.
Editorial extensions
If this is right
- Adding the relevance-check and query-refinement stage raises the average human evaluation score and the mean cosine similarity for all three generation models tested, while also lowering the spread of cosine-similarity scores.
- The advanced pipeline yields higher mean cosine similarity than vanilla RAG in every tested question category, including factual questions, temporal-change questions, statistical questions, Bangla dialect questions, and grammar or spelling-error questions.
- Out-of-context questions remain the weakest category for both pipelines, so a RAG system alone does not reliably detect when a query falls outside the document collection.
- Lowering sampling temperature to 0.1 gives the highest mean cosine similarity for all three generation models, consistent with legal question answering needing exact answers rather than creative variation.
- Using Bangla or English for the instruction prompt makes little difference to the scores, suggesting the multilingual embedding keeps the pipeline robust to prompt-language variation.
Reading between the lines
- The same two-stage filter could plausibly transfer to other code-mixed official records beyond police gazettes, but the paper does not test that transfer.
- Because the question-answer pairs were drafted from the same gazette text the system retrieves from, the benchmark may under-represent how real users phrase legal queries; a separate user study would test that gap.
- The reported aggregate scores do not separate gains from discarding irrelevant chunks from gains due to query rephrasing; logging retrieval decisions per query would settle which mechanism carries the improvement.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents LegalRAG, a bilingual retrieval-augmented generation system for question answering on Bangladesh Police Gazettes, which contain mixed English and Bangla content. The authors build a vanilla RAG pipeline (bge-m3 embeddings, ChromaDB, MMR retrieval, and a generator LLM) and an advanced RAG pipeline that inserts a Llama 3.2 (3B) model to check whether retrieved chunks are relevant and, if not, to refine the query iteratively. They evaluate both pipelines on a self-curated set of 168 GPT-4o-generated Bangla question-answer pairs using mean cosine similarity and average human scores from three legal-domain evaluators. The central empirical claim is that the advanced pipeline outperforms the vanilla pipeline on this test set, which is supported by Tables IV and V, while the abstract makes the broader and less supported claim that the approach 'consistently outperforms existing methods across all evaluation metrics.'
Significance. If the narrow result holds, this is a modest but useful proof-of-concept: a lightweight relevance-check and query-refinement stage improves retrieval and answer quality for low-resource bilingual legal documents. The paper's strengths are its focus on a genuinely low-resource domain (Bangla legal gazettes), the inclusion of realistic query types such as dialectal and misspelled inputs, the use of both automatic and human evaluation, and ablations of temperature and prompt language. The significance is limited, however, by the internally generated benchmark, the absence of any non-vanilla RAG baselines, the lack of statistical testing or inter-annotator agreement, and the unverified OCR quality that feeds every evaluation component. No code, data, or OCR output is released, so the results are not independently reproducible as reported.
major comments (5)
- [Section III.B (Data Preprocessing)] The OCR stage is not validated in any quantitative way. The paper states only that 'Tesseract OCR is configured to recognize both Bangla and English texts from the images' and reports no character error rate, word error rate, per-language accuracy, or manual spot-check. Because the same OCR output is used to build the vector index, to generate the 168 ground-truth QA pairs with GPT-4o, and as the reference for semantic similarity and human evaluation, any systematic corruption of Bangla legal terminology propagates into every component. The vanilla-versus-advanced comparison is not automatically invalid, but the headline claim is about answering questions on legal documents, not on OCR artifacts; the authors should add an OCR accuracy assessment on a held-out page sample and report language-specific error rates.
- [Abstract and Sections IV.A/IV.B] The abstract claims that 'our approach consistently outperforms existing methods across all evaluation metrics,' but the experiments compare only the proposed Advanced RAG against a single Vanilla RAG baseline on a self-constructed test set. None of the RAG variants discussed in Section II (e.g., Self-RAG, query-document alignment, or Blended RAG) is used as a baseline, and no public benchmark is used. Either the claim should be restricted to 'outperforms the vanilla RAG pipeline on this test set,' or the required baselines and standard datasets should be added.
- [Section IV.B, Tables IV and V] No statistical significance is reported for any comparison in Tables IV and V. The differences in human scores (e.g., Llama 3.1 8B: 3.41 vs. 3.70) and cosine similarity (0.76 vs. 0.82) are given as point estimates without confidence intervals, paired tests, or per-question variance. Given the small test set (n=168) and the multiple comparisons across three LLMs and eight domains, the observed differences could be within sampling noise. The authors should provide paired significance tests (e.g., bootstrap or permutation tests) and effect sizes, or explicitly label the comparisons as descriptive only.
- [Section IV.C and Figure 5(a)] The sampling temperature is selected on the same test set used for final evaluation. Figure 5(a) reports mean cosine similarity for temperatures 0.1, 0.4, 0.7, and 1.0, and the paper then uses 0.1 for all reported results, selecting the best value on the test set. This is a form of test-set overfitting. There is no validation/tuning split, and other fixed hyperparameters (retrieval chunk count k, maximum refinement iterations, MMR settings) are not varied. The authors should either use a development/test split and choose hyperparameters on the development split, or report the sensitivity of the main conclusions to these choices.
- [Section IV.A, Evaluation, and Table III] The human evaluation reports average scores from three evaluators but provides no inter-annotator agreement statistic. Since the ratings are subjective and the pipeline differences are small (at most 0.4 on a 1-5 scale), an agreement measure such as Krippendorff's alpha or quadratic weighted kappa is needed to establish that the signal is not driven by a single rater. In addition, the paper does not state whether evaluators were blinded to which pipeline generated each response; this should be clarified.
minor comments (5)
- [Section III.C, Table II] The claim that vanilla RAG is acceptable for science and finance domains is supported only by Table II, but no details are given for how those semantic similarity numbers were computed or on what data; this table should either be moved to an appendix with full methodology or removed.
- [Section III.B, step 5] The sentence 'The retriever then measured the similarity' mixes tenses; it should read 'The retriever then measures the similarity.'
- [Figure 1] Figure 1 is dense and the relationship between the generated answers, the relevance labels, and the retrieved chunks is hard to follow; a simplified diagram or a table with aligned rows would improve readability.
- [References] Reference formatting is inconsistent: arXiv identifiers, access dates, and conference styles are mixed, and at least one reference (Ref. [5]) duplicates the same work as Ref. [2]; the list should be normalized.
- [Ethics Statement] The ethics statement says that 'no personal or sensitive data were collected or stored,' yet Table III reports age and gender summaries; the authors should clarify that only de-identified demographic summaries were retained.
Circularity Check
No construction-level circularity found: the Advanced RAG comparison is an empirical evaluation, and no prediction or derivation reduces to its own inputs by construction.
full rationale
The paper makes no formal derivation that could collapse into its inputs. Its central claim is an empirical comparison between a Vanilla RAG pipeline and an Advanced RAG pipeline that adds a relevance-check/query-refinement LLM stage. The reported metrics are human ratings and cosine similarity against a curated ground truth; neither metric is defined in terms of the Advanced RAG output in a way that would force the observed improvement. The test set is generated from the same OCR-derived source text that the system indexes, but this is standard closed-domain QA construction rather than circularity: both compared pipelines face the same corpus and the same ground truth, so the Vanilla-versus-Advanced contrast remains meaningful. The lack of OCR accuracy verification and the use of author-affiliated evaluators are validity and generalizability risks, not construction circularity. Reference [16] includes a co-author only in an editorial capacity and is used as background motivation, not as a load-bearing uniqueness or correctness argument. The temperature of 0.1 is stated in the experimental setup before the reported results and is then examined in an ablation; even if the ablation indicated that this temperature performed best on the test set, that would be evaluation leakage rather than an equation-level equivalence. Thus, no specific circular step satisfying the evidence standard can be identified.
Assumptions & free parameters
free parameters (3)
- Maximum refinement iterations =
3
- Retrieval chunk count k =
3 (appears in Figure 1 caption)
- Sampling temperature =
0.1
assumptions (4)
- domain assumption BGE-M3 embeddings preserve semantic similarity across Bangla and English sufficiently for retrieval.
- domain assumption GPT-4o generated question-answer pairs form accurate ground truth for legal QA.
- domain assumption Cosine similarity between response and ground truth embeddings is a valid correctness measure.
- domain assumption Three legal-trained native evaluators give reliable human scores without measurable agreement.
Cite this review
Pith. "Pith review of LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval." pith.science (2026). https://pith.science/paper/QR57GND3
@misc{pith2026250416121,
author = {Pith},
title = {Pith review of: LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/QR57GND3}},
note = {Machine review of arXiv:2504.16121}
}
read the original abstract
Natural Language Processing (NLP) and computational linguistic techniques are increasingly being applied across various domains, yet their use in legal and regulatory tasks remains limited. To address this gap, we develop an efficient bilingual question-answering framework for regulatory documents, specifically the Bangladesh Police Gazettes, which contain both English and Bangla text. Our approach employs modern Retrieval Augmented Generation (RAG) pipelines to enhance information retrieval and response generation. In addition to conventional RAG pipelines, we propose an advanced RAG-based approach that improves retrieval performance, leading to more precise answers. This system enables efficient searching for specific government legal notices, making legal information more accessible. We evaluate both our proposed and conventional RAG systems on a diverse test set on Bangladesh Police Gazettes, demonstrating that our approach consistently outperforms existing methods across all evaluation metrics.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Natural language processing,
K. Chowdhary and K. Chowdhary, “Natural language processing,” Fundamentals of artificial intelligence , pp. 603–649, 2020
2020
-
[2]
Optical character recognition by open source ocr tool tesseract: A case study,
C. Patel, A. Patel, and D. Patel, “Optical character recognition by open source ocr tool tesseract: A case study,” International journal of computer applications, vol. 55, no. 10, pp. 50–56, 2012
work page 2012
-
[3]
S. Kawashima, S. Shiramatsu, and T. Mizumoto, “Development of rag system for digital transformation of local government and considering optimal document-segmentation methods,” in International Congress on Information and Communication Technology. Springer, 2024, pp. 603– 617
work page 2024
-
[4]
Nlp-based automated compliance checking of data processing agreements against gdpr,
O. A. Cejas, M. I. Azeem, S. Abualhaija, and L. C. Briand, “Nlp-based automated compliance checking of data processing agreements against gdpr,” IEEE Transactions on Software Engineering , vol. 49, no. 9, pp. 4282–4303, 2023
work page 2023
-
[5]
Optical character recognition by open source ocr tool tesseract: A case study,
D. P. Chirag Patel, Atul Patel, “Optical character recognition by open source ocr tool tesseract: A case study,” International Journal of Computer Applications , vol. 55, no. 10, pp. 50–56, October 2012. [Online]. Available: https://ijcaonline.org/archives/volume55/number10/ 8794-2784/
work page 2012
-
[6]
Chatgpt: Unlocking the future of nlp in finance,
A. Zaremba and E. Demir, “Chatgpt: Unlocking the future of nlp in finance,” Modern Finance, vol. 1, no. 1, pp. 93–98, 2023
work page 2023
-
[7]
Retrieval- augmented generation for knowledge-intensive nlp tasks,
P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020
2020
-
[8]
Natural language processing in the legal domain,
D. M. Katz, D. Hartung, L. Gerlach, A. Jana, and M. J. Bommarito II, “Natural language processing in the legal domain,” arXiv preprint arXiv:2302.12039, 2023
arXiv 2023
Show all 37 references
-
[9]
How does NLP benefit legal system: A summary of legal artificial intelligence,
H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun, “How does NLP benefit legal system: A summary of legal artificial intelligence,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , D. Jurafsky, J. Chai, N. Schluter, and J. Tetre...
2020
-
[10]
Document analysis,
G. Davie and D. Wyatt, “Document analysis,” in The Routledge hand- book of research methods in the study of religion . Routledge, 2021, pp. 245–255
2021
-
[11]
Information retrieval: recent advances and beyond,
K. A. Hambarde and H. Proenca, “Information retrieval: recent advances and beyond,” IEEE Access, 2023
2023
-
[12]
LexDrafter: Terminology drafting for legislative documents using retrieval augmented generation,
A. Chouhan and M. Gertz, “LexDrafter: Terminology drafting for legislative documents using retrieval augmented generation,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , N. Calzola...
2024
-
[13]
Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain,
N. Pipitone and G. H. Alami, “Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain,” arXiv preprint arXiv:2408.10343, 2024
2024 arXiv
-
[14]
Benchmarking large language models in retrieval-augmented generation,
J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking large language models in retrieval-augmented generation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 16, 2024, pp. 17 754– 17 762
2024
-
[15]
Retrieval-augmented generation in multilingual settings,
N. Chirkova, D. Rau, H. D ´ejean, T. Formal, S. Clinchant, and V . Nikoulina, “Retrieval-augmented generation in multilingual settings,” arXiv preprint arXiv:2407.01463 , 2024
2024 arXiv
-
[16]
Crosslingual retrieval augmented in-context learning for Bangla,
X. Li, E. Nie, and S. Liang, “Crosslingual retrieval augmented in-context learning for Bangla,” in Proceedings of the First Workshop on Bangla Language Processing (BLP-2023) , F. Alam, S. Kar, S. A. Chowdhury, F. Sadeque, and R. Amin, Eds. Singapore: Association for Computatio...
2023
-
[17]
The llama 3 herd of models,
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024
2024 arXiv
-
[18]
Gpt-4 technical report,
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
2023 arXiv
-
[19]
Self-RAG: Learning to retrieve, generate, and critique through self-reflection,
A. Asai, Z. Wu, Y . Wang, A. Sil, and H. Hajishirzi, “Self-RAG: Learning to retrieve, generate, and critique through self-reflection,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=hSyW5go0v8
2024
-
[20]
Making retrieval- augmented language models robust to irrelevant context,
O. Yoran, T. Wolfson, O. Ram, and J. Berant, “Making retrieval- augmented language models robust to irrelevant context,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=ZS4m74kZpH
2024
-
[21]
Optimizing query generation for enhanced document retrieval in rag,
H. Koo, M. Kim, and S. J. Hwang, “Optimizing query generation for enhanced document retrieval in rag,” arXiv preprint arXiv:2407.12325 , 2024
2024 arXiv
-
[22]
Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrievers,
K. Sawarkar, A. Mangal, and S. R. Solanki, “Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrievers,” arXiv preprint arXiv:2404.07220 , 2024
2024 arXiv
-
[23]
Gpt-4o system card,
A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024
2024 arXiv
-
[24]
Retrieval augmentation reduces hallucination in conversation,
K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” in Findings of the Association for Computational Linguistics: EMNLP 2021 , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Punta Cana, Dominican Repu...
2021
-
[25]
Reducing hallucination in structured outputs via retrieval-augmented generation,
O. Ayala and P. Bechard, “Reducing hallucination in structured outputs via retrieval-augmented generation,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Tra...
2024
-
[26]
Mixtral of experts,
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al. , “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024
2024 arXiv
-
[27]
Gemma 2: Improving open language models at a practical size,
G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram ´e et al. , “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024
2024 arXiv
-
[28]
Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,
J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,” 2024
2024
-
[29]
An overview of the tesseract ocr engine,
R. Smith, “An overview of the tesseract ocr engine,” in Ninth Inter- national Conference on Document Analysis and Recognition (ICDAR 2007), vol. 2, 2007, pp. 629–633
2007
-
[30]
QLoRA: Efficient finetuning of quantized LLMs,
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=OUIFPHEgJU
2023
-
[31]
The effect of sampling temperature on problem solving in large language models,
M. Renze, “The effect of sampling temperature on problem solving in large language models,” in Findings of the Association for Computational Linguistics: EMNLP 2024 , Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguistic...
2024
-
[32]
Harnessing legal complex- ity,
J. Ruhl, D. M. Katz, and M. J. Bommarito, “Harnessing legal complex- ity,” Science, vol. 355, no. 6332, pp. 1377–1378, 2017
2017
-
[33]
Harnessing the complexity of legal systems for governing global challenges,
J. Ruhl and D. M. Katz, “Harnessing the complexity of legal systems for governing global challenges,” Global challenges, governance, and complexity: Applications and frontiers , pp. 147–165, 2019
2019
-
[34]
Computational methods in legal analysis,
J. Frankenreiter and M. A. Livermore, “Computational methods in legal analysis,” Annual Review of Law and Social Science , vol. 16, no. 1, pp. 39–57, 2020
2020
-
[35]
On the concept of relevance in legal information retrieval,
M. Van Opijnen and C. Santos, “On the concept of relevance in legal information retrieval,” Artificial Intelligence and Law , vol. 25, pp. 65– 87, 2017
2017
-
[36]
Legal information retrieval systems: State- of-the-art and open issues,
C. Sansone and G. Sperl ´ı, “Legal information retrieval systems: State- of-the-art and open issues,” Information Systems , vol. 106, p. 101967, 2022
2022
-
[37]
Legal information retrieval for understand- ing statutory terms,
J. ˇSavelka and K. D. Ashley, “Legal information retrieval for understand- ing statutory terms,” Artificial Intelligence and Law , pp. 1–45, 2022
2022
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.