{"id":"9f2bad94-18fb-4125-8412-2e6e949daa14","arxiv_id":"2606.07523","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"A RAG pipeline using BM25 retrieval and GPT-o3 generation achieves 91% Precision@1 and 85% truthfulness for Nepali legal question answering on a curated 100-query benchmark.","lead":"This paper builds a RAG pipeline for answering legal questions in Nepali using Supreme Court case law, finding that BM25 sparse retrieval outperforms dense embeddings. A smart generalist might read it to understand how standard NLP tools can be adapted for low-resource legal domains where fine-tuning is impractical.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Expert-authored queries may systematically favor BM25 over dense retrieval by mirroring document vocabulary, inflating the headline BM25-vs-dense gap.","rationale":"The reader correctly identified the 100-query test set as a limitation but framed the concern primarily around sample size and single-ground-truth design affecting generalizability. My concern is more specific: the query authoring methodology (domain experts with document access) creates a structural bias in the retrieval comparison itself, not just a generalizability issue. However, this does not change the verdict from CONDITIONAL. The paper remains a legitimate first-application contribution to an under-resourced domain with shipped code and data. The BM25 > dense finding is consistent with prior work (the paper cites [16], [5] in support), and even under the paraphrase test, BM25 would likely still perform reasonably — just perhaps not with a 16-point gap. The concern strengthens the case for the reader's recommended improvements (larger test set, significance testing) but does not constitute a load-bearing flaw that would move the verdict to REJECT. The generation evaluation concerns (LLM-as-judge, no IAA) are real but secondary to the retrieval comparison, which is the paper's primary contribution.","tokens_in":13133,"tokens_out":1826,"duration_ms":49601,"concrete_test":"Have 3–5 non-lawyer Nepali speakers paraphrase the 100 queries in plain language without seeing the source documents. Re-run all five retrieval methods on the paraphrased queries using the same ground-truth mappings. If BM25's P@1 drops by more than ~10 points relative to e5-large's P@1 (i.e., the gap narrows from 16 to under ~6 points), the BM25 superiority claim is substantially an artifact of expert query formulation rather than a domain property.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that BM25 substantially outperforms dense retrieval for Nepali legal text (P@1 0.91 vs 0.75) — depends on the 100-query test set. These queries were 'deliberately curated by legal domain scholars' (§III.A) who had access to the source documents. Legal experts naturally formulate queries using the exact statutory terminology and formulaic phrasing found in the documents themselves. This creates a systematic lexical-overlap bias that advantages BM25 (term-frequency matching) over dense retrievers (semantic matching). The paper acknowledges that Nepali legal text is 'lexically constrained and highly formulaic' (§V) but frames this as a domain property rather than a potential artifact of the query construction methodology. If the queries were paraphrased in natural, non-expert language — as a layperson seeking legal information would actually phrase them — the BM25 advantage could shrink substantially, since dense retrievers are designed to bridge vocabulary gaps that BM25 cannot. The paper's own framing ('public engagement in the legal domain,' §I) implies the system should serve non-expert users, yet the evaluation uses expert-formulated queries that structurally mirror document language. This is more specific than the generic small-sample concern: it directly threatens the headline finding's validity for the intended use case.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper presents the first RAG-based question-answering pipeline for the Nepali legal domain, using Supreme Court case law documents from the Nepal Kanun Patrika (NKP) digital archive. The authors compare sparse retrieval (BM25 at document and chunk level) against dense retrieval (multilingual-e5-large, multilingual-e5-base, LaBSE) on a 100-query expert-annotated test set, and evaluate end-to-end generation using GPT-o3 with a two-step evidence-extraction prompt. The headline finding is that BM25 on chunked documents achieves Precision@1 of 0.91, substantially outperforming the best dense retriever (multilingual-e5-large at 0.75), and that the full pipeline achieves 85% truthfulness and 74% groundedness. The dataset (10,265 documents, 110,626 chunks) and code are publicly available, which is a notable strength for reproducibility in a low-resource setting.","tokens_in":14028,"tokens_out":1571,"duration_ms":135048,"significance":"The paper addresses a genuine gap: there is no prior systematic evaluation of retrieval strategies for Nepali legal QA, and the curated corpus of Supreme Court case law is a contribution in itself. The public release of both dataset and code is commendable and strengthens reproducibility. The finding that BM25 outperforms off-the-shelf dense retrievers in this domain is consistent with prior work on formulaic legal text [16, 21] and provides a useful baseline for future research. However, the significance of the headline BM25-vs-dense comparison is tempered by the evaluation methodology concerns detailed below.","major_comments":[{"comment":"§III.A and §V: The 100-query test set is constructed by legal domain experts who had access to the source documents, and each query maps to exactly one ground-truth document. The paper itself notes that Nepali legal text is 'lexically constrained and highly formulaic' (§V), which it frames as a domain property explaining BM25's advantage. However, this framing overlooks a methodological confound: expert-authored queries likely mirror the exact statutory terminology and formulaic phrasing of the source documents, systematically favoring BM25's term-frequency matching over dense semantic retrieval. The paper's stated goal of serving 'public engagement in the legal domain' (§I) implies non-expert users, yet the evaluation uses expert-formulated queries that structurally resemble document language. This is a load-bearing concern because the headline finding (P@1: 0.91 vs 0.75) may not hold —","section":null},{"comment":"§IV.B and Table V: The generation evaluation reports truthfulness (85%), groundedness (74%), and successful answer rate (92%) for BM25_Docs, but these are computed on the same 100-query test set used for retrieval evaluation with no held-out split or cross-validation. The paper acknowledges this limitation in §VI ('hinders broad statistical generalization'), but the generation quality claims — particularly the comparison between BM25_Docs, e5-large, and the Count method — rest entirely on this single evaluation. A different query set could shift the relative ranking of generation methods. At minimum, the authors should report confidence intervals or bootstrap estimates for the generation metrics in Table V to indicate the uncertainty introduced by the small sample.","section":null},{"comment":"§III.E and Appendix C: The groundedness evaluation uses Gemini 2.5 Pro as an LLM-as-judge with binary (0/1) scoring, but the paper does not report any validation of the judge's reliability — no inter-annotator agreement with the human evaluators, no calibration against a gold-standard subset. Given that the 74% groundedness figure is a headline result, the absence of judge validation is a gap. The human evaluation (84% truthfulness) is compared to the LLM judge (85% truthfulness), but the paper does not report whether these were on the same queries or how agreement was computed. Please clarify the relationship between the human and automated evaluations and report agreement statistics.","section":null}],"minor_comments":[{"comment":"The model referred to as 'GPT-o3' or 'OpenAI's 03 model' (§III.D, §V) appears to reference OpenAI's o3 model, but the citation [4] (Cao et al., 2025) is about retrieval robustness evaluation, not the o3 model itself. Please cite the appropriate technical report for the model used.","section":null},{"comment":"Table IV (latency comparison) reports BM25_Chunks at 1.2003s/query and BM25_Docs at 0.1042s/query, but the text in §IV.A states the latency difference is '11 times higher.' The ratio is approximately 11.5, which is close, but the text should state the precise figure or round consistently.","section":null},{"comment":"§III.D: The 'Count Method' is described as selecting 'the most frequent document after aggregating the top 5 documents retrieved by BM25_Docs, BM25_Chunks, and e5-large.' It is unclear whether this means the top-5 from each method (15 total) or the top-5 overall. Please clarify the aggregation procedure.","section":null},{"comment":"The document text contains numerous OCR or encoding artifacts (e.g., 'Bal Kr 1108. 381' in the author list, 'Kathmandu Universit;' in the affiliation, 'Fmail' instead of 'Email', 'tem' instead of 'system' in §I). These should be corrected for the camera-ready version.","section":null},{"comment":"§III.A: The test set composition is described as 79% objective (87% fact/definition, 13% party identification) and 21% subjective (58% procedure, 42% rationale). Please include example queries from each category in an appendix or supplementary material to help readers assess query difficulty and lexical overlap with source documents.","section":null},{"comment":"Reference [16] (Mori et al., 2025) is cited as supporting the finding that BM25 outperforms dense retrieval in formulaic legal text. This appears to be a preprint (arXiv:2506.12895). Please verify whether it has been peer-reviewed and cite its preprint status.","section":null},{"comment":"§V: The paper states 'BM25_Chunks provided 8 slight performance improvement of 3.4%' over BM25_Docs. The '8' appears to be a typo for 'a'. Also, the 3.4% figure should be verified: 0.91 vs 0.88 is a 3 percentage-point difference, which is a 3.4% relative improvement — please state whether this is absolute or relative.","section":null},{"comment":"The paper does not report any statistical significance tests for the retrieval performance differences in Table III. Given the small test set (n=100), reporting whether the BM25 vs e5-large difference is statistically significant (e.g., via a paired test) would strengthen the claims.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about expert-authored queries favoring BM25 is, in my assessment, the most important issue for this paper. It is more specific than the generic small-sample concern and directly threatens the validity of the headline finding for the paper's stated use case (public engagement). The authors should be asked to address this directly, either by testing with paraphrased/layperson queries or by explicitly scoping their claims to expert-formulated factual lookup queries. The paper is otherwise a solid first baseline for an under-researched domain, and the public data/code release is a genuine contribution."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee raises three major concerns: (1) expert-authored queries may systematically favor BM25 over dense retrievers, confounding the headline comparison; (2) generation metrics lack confidence intervals or cross-validation on the 100-query test set; and (3) the LLM-as-judge (Gemini 2.5 Pro) for groundedness lacks validation against human annotators. We agree that all three points identify genuine methodological gaps. We will address (2) and (3) directly by adding bootstrap confidence intervals and inter-annotator agreement statistics in the revision. For (1), we partially agree—the expert-query confound is real—but we explain why the finding remains informative as a baseline while acknowledging the limitation more explicitly. We cannot fully resolve (1) without collecting a non-expert query set, which is beyond the scope of a revision but is planned as future work.","responses":[{"response":"The referee raises a valid and important methodological concern. We agree that expert-authored queries are more likely to mirror the lexical and formulaic phrasing of the source documents, which can systematically favor BM25's term-frequency matching. This is a genuine confound in our experimental design. We will revise the manuscript to acknowledge this limitation explicitly in both the Discussion (§V) and the methodology description (§III.A), rather than framing the BM25 advantage solely as a domain property. That said, we note two points that partially mitigate the concern. First, our finding is consistent with prior work on formulaic legal text in other languages (refs [16, 21, 25] in the manuscript), where BM25 advantages were observed on non-expert-constructed benchmarks as well. Second, 21% of our test queries are subjective (assessing legal procedure and reasoning), which are less likely to directly mirror document terminology. However, we acknowledge that 79% objective queries may still exhibit lexical overlap bias even when not directly quoting statutory language. We cannot fully resolve this confound within the scope of a revision—collecting a non-expert query set with verified ground-truth annotations requires substantial expert validation, which we have planned as future work. We will add this as an explicit limitation and adjust our claims to scope the BM25-vs-dense comparison to expert-formulated queries rather than general user queries.","revision_made":"partial","referee_comment":"Expert-authored queries likely mirror the exact statutory terminology and formulaic phrasing of the source documents, systematically favoring BM25 over dense retrieval. The stated goal of serving public engagement implies non-expert users, yet the evaluation uses expert-formulated queries."},{"response":"We agree. The 100-query test set is small, and reporting point estimates without uncertainty intervals overstates the precision of the generation metrics. We will add bootstrap 95% confidence intervals for all metrics in Table V (truthfulness, groundedness, and successful answer rate) across all three methods (BM25_Docs, e5-large, Count). We will also clarify in §IV.B that the retrieval and generation evaluations use the same 100-query set—there is no separate held-out split—and explain why this was necessary given the cost and expertise required for expert annotation of legal ground-truth answers. We acknowledge that a different query set could shift the relative ranking of generation methods, and the confidence intervals will make this uncertainty visible to readers. We will also note in the revision that the comparison between BM25_Docs and e5-large (85% vs 78% truthfulness) should be interpreted in light of the overlapping or non-overlapping confidence intervals once computed.","revision_made":"yes","referee_comment":"Generation metrics (truthfulness, groundedness, successful answer rate) are computed on the same 100-query test set with no held-out split or cross-validation. Confidence intervals or bootstrap estimates should be reported for Table V."},{"response":"We agree that this is a gap. The LLM-as-judge evaluation lacks validation against human annotations, and the relationship between the human truthfulness evaluation (84%) and the automated truthfulness evaluation (85%) is not clearly specified. We will address this in the revision as follows. First, we will clarify that the human and automated truthfulness evaluations were conducted on the same 100 queries for BM25_Docs, and we will report inter-annotator agreement (Cohen's kappa or percentage agreement) between the human evaluators and the Gemini 2.5 Pro judge on the truthfulness dimension. Second, we will create a gold-standard subset (approximately 30-50 queries) where human annotators also perform the groundedness evaluation, and report agreement statistics between the human groundedness judgments and the LLM judge's binary groundedness scores. This will provide calibration for the 74% groundedness headline figure. We acknowledge that if agreement is low, the groundedness number should be interpreted with caution. We will report these results transparently regardless of the outcome.","revision_made":"yes","referee_comment":"The groundedness evaluation uses Gemini 2.5 Pro as an LLM-as-judge with binary scoring, but no validation of the judge's reliability is reported—no inter-annotator agreement with human evaluators, no calibration against a gold-standard subset. The relationship between human (84% truthfulness) and automated (85% truthfulness) evaluations is unclear."}],"tokens_in":13067,"tokens_out":1096,"duration_ms":109865,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The paper you should know about: this is the first RAG pipeline applied to Nepali legal case law, with a curated corpus of ~10K Supreme Court documents from the Nepal Kanun Patrika archive and a 100-query expert-annotated benchmark. The headline finding is that BM25 substantially outperforms dense retrievers (P@1 of 0.91 vs 0.75 for multilingual-e5-large), which aligns with prior work on formulaic legal text. Code and data are public on GitHub. The contribution is the domain application and the dataset, not the methods — BM25, FAISS, LangChain, GPT-o3 are all off-the-shelf. That's fine for what this is. They also ship their prompts in an appendix, which I appreciate. The two-step evidence-extraction-then-generation prompting is a sensible design choice and they report both LLM-as-judge and human evaluation, which is more than many papers in this space do. The finding that retrieval quality drives generation quality is not surprising but is cleanly demonstrated. They also make a reasonable latency tradeoff argument for preferring BM25_Docs over BM25_Chunks. Now the soft spots. The stress-test concern about expert-authored queries is the one I'd press hardest. Legal scholars who have read the source documents will naturally formulate queries using the exact statutory terminology found in those documents. This creates systematic lexical overlap that advantages BM25 over dense retrievers — not because BM25 is better for Nepali legal text in general, but because the queries structurally mirror the documents. The paper acknowledges that Nepali legal text is ","headline":"First RAG pipeline for Nepali legal QA — legitimate domain contribution, standard methods, evaluation has a real methodological gap worth flagging","tokens_in":13916,"tokens_out":1071,"would_cite":false,"duration_ms":38630,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Simple keyword search beats AI embeddings for Nepali legal documents","keywords":[],"falsifier":"Construct a larger test set (say 500+ queries) of Nepali legal questions where multiple documents may be relevant, and re-run all five retrieval methods. If BM25's Precision@1 drops substantially or if dense retrieval closes the gap, the central claim that sparse retrieval is inherently superior for Nepali legal text would be weakened.","tokens_in":13384,"feed_emoji":"⚖️","tokens_out":1112,"duration_ms":113890,"temperature":0.7,"pith_summary":"This paper builds the first retrieval-augmented generation (RAG) pipeline for answering legal questions in Nepali, a low-resource language with no existing legal NLP benchmarks. The authors scrape 10,265 Supreme Court case documents from the Nepal Kanun Patrika digital archive, chunk them into 500-token passages, and compare five retrieval methods against 100 expert-annotated queries. The central finding is that BM25 — a decades-old term-frequency ranking algorithm — retrieves the correct document 91% of the time at rank one on chunked text and 88% on full documents, while the best dense embedding model (multilingual-e5-large) manages only 75%. The authors attribute this gap to the formulaic, lexically rigid nature of Nepali legal prose, where exact statutory phrasing makes keyword overlap more informative than semantic approximation. When paired with GPT-o3 using a two-step prompt that first extracts verbatim legal evidence then generates an answer, the BM25 pipeline achieves 85% truthfulness and 74% groundedness on the benchmark. The paper argues that for low-resource legal domains, a well-structured retrieval pipeline with classical search can substitute for the fine-tuned legal LLMs that data scarcity makes infeasible.","feed_headline":"Old-school keyword search beats AI for Nepali legal docs","feed_subtitle":"BM25 retrieval hits 91% precision on Supreme Court cases, outperforming dense embeddings by 16 points and grounding an 85%-truthful QA bot.","key_machinery":"The pipeline has three load-bearing components. First, BM25 retrieval on the full document corpus, which ranks documents by term frequency and document length normalization. Second, a two-step prompting strategy where the language model first extracts verbatim legal sentences from the retrieved context as evidence, then generates a final answer grounded only in that evidence, with an explicit refusal instruction when evidence is insufficient. Third, an LLM-as-judge evaluation using binary yes/no prompts for groundedness (all claims verifiable in source) and truthfulness (semantic equivalence with a reference answer), cross-checked against human evaluation. The corpus itself is a contribution","core_discovery":"BM25 sparse retrieval substantially outperforms multilingual dense embedding models on Nepali legal text, achieving Precision@1 of 0.91 on chunked documents versus 0.75 for the best dense retriever, because the formulaic and lexically constrained nature of legal language makes exact term matching more effective than semantic approximation. This retrieval quality directly determines generation quality: when BM25-retrieved context feeds a two-step evidence-extraction prompt with GPT-o3, the pipeline produces 85% truthful and 74% grounded answers, establishing a working baseline for legal QA in a language with no prior legal NLP infrastructure.","pith_inferences":["The 100-query evaluation set with single ground-truth documents per query means the BM25 advantage may shrink on multi-document or ambiguous queries where semantic matching could help; the paper's conclusion that sparse retrieval is inherently better for this domain should be tested on a larger, multi-relevance benchmark.","The refusal strategy (answering only when sufficient evidence is present) means the 92% successful answer generation rate also functions as a recall ceiling; the 8% of refused queries represent cases where retrieval failed, and analyzing those failure modes could reveal whether dense retrieval complements BM25 on the queries it misses.","The latency difference between BM25 on chunks (1.2s) versus full documents (0.1s) suggests a hybrid approach using full-document BM25 for initial filtering followed by chunk-level re-ranking on top candidates could capture most of the accuracy gain at a fraction of the latency cost."],"forward_implications":["For other low-resource languages with formulaic legal traditions, BM25 may remain competitive with or superior to dense retrieval, reducing the barrier to building legal QA systems without large annotated training sets.","The two-step evidence-extraction-then-generation prompt pattern could be adopted in other high-stakes domains where hallucination risk must be minimized, as it forces verifiable grounding before answer synthesis.","The publicly released dataset of 10,265 Nepali Supreme Court case documents creates a foundation for future legal NLP benchmarks in Nepali, including hybrid retrieval, re-ranking, and fine-tuning experiments.","The gap between BM25 and dense retrieval on legal text suggests that domain-adaptive pretraining of multilingual embeddings on legal corpora could narrow the performance difference, pointing to a concrete training target."],"fun_headline_variants":["BM25 beats dense embeddings for Nepali legal QA retrieval","Keyword search hits 91% precision on Nepali Supreme Court cases","Sparse retrieval outperforms multilingual E5 on Nepali legal text","RAG pipeline for Nepali law relies on BM25 over dense embeddings","Exact term matching wins for formulaic Nepali legal language"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire performance comparison rests on 100 expert-annotated queries, each mapped to exactly one ground-truth document, with no held-out validation or cross-validation. A different or larger query set could shift the reported gap between BM25 and dense retrieval, and the single-document-per-query design does not reflect realistic legal QA where multiple documents may be relevant.","fun_headline_variants_meta":{"raw":{"variants":["BM25 beats dense embeddings for Nepali legal QA retrieval","Keyword search hits 91% precision on Nepali Supreme Court cases","Sparse retrieval outperforms multilingual E5 on Nepali legal text","RAG pipeline for Nepali law relies on BM25 over dense embeddings","Exact term matching wins for formulaic Nepali legal language"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":620,"prompt_tokens":532,"completion_tokens":88,"prompt_tokens_details":null},"tokens_in":532,"tokens_out":88,"duration_ms":15963,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-05T15:51:41.956750+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Construct a larger test set (say 500+ queries) of Nepali legal questions where multiple documents may be relevant, and re-run all five retrieval methods. If BM25's Precision@1 drops substantially or if dense retrieval closes the gap, the central claim that sparse retrieval is inherently superior for Nepali legal text would be weakened.","supporting_citations":[],"review_version":1}