{"id":"61e946be-e015-466b-9988-c382682d4fe1","arxiv_id":"2608.12820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BGE-M3 cross-lingual embeddings achieve higher retrieval recall than query translation for Sinhala-English and Tamil-English government document search.","lead":"This paper compares two ways to let Sinhala and Tamil speakers search English government documents: translating the question first, or using multilingual embedding models that align languages directly. In tests on 500 Sri Lankan government queries, the BGE-M3 embedding model beat all translation-based approaches, reaching about 96% recall at rank 15.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The QT vs CLE comparison is confounded: QT always uses FastEmbed, while CLE-BGE-M3 uses BGE-M3 for both query and documents, so the headline gap may reflect encoder quality rather than the cross-lingual paradigm.","rationale":"The Reader's weakest_assumption identifies exactly the load-bearing concern: QT and CLE pipelines differ not only in the translation step but in the retrieval encoder, and the paper does not acknowledge this. The central empirical claim that cross-lingual embeddings are more effective than query translation is therefore not established as stated. I grant the paper real strengths: a human-verified Sinhala/Tamil/English benchmark from a real government corpus, a clear three-pipeline setup, and a credible demonstration that monolingual retrieval fails badly. Those contributions justify publication conditional on a matched-encoder control. The fix is inexpensive and does not require new data — only re-running the existing translations with BGE-M3 — so rejection is not warranted. This reaffirms the Reader's CONDITIONAL verdict rather than changing it. The final recommendation is CONDITIONAL: accept after the authors run the QT-BGE-M3 control and release the benchmark artifacts.","tokens_in":7147,"tokens_out":8035,"duration_ms":78031,"concrete_test":"Re-run the QT-GT pipeline with BGE-M3 as the embedding model: keep the same 500 verified Sinhala/Tamil queries, the same Google Translate English outputs, and the same 1,699 English contexts, but embed both the translated queries and the documents with BGE-M3 (instead of FastEmbed), then recompute Recall@15. If QT-GT+BGE-M3 reaches or exceeds CLE-BGE-M3 (96.2%/95.6%), the claimed advantage of cross-lingual embeddings over query translation is an artifact of encoder choice; if CLE-BGE-M3 still wins by a similar margin, the confound is refuted. A useful supplementary check is to embed the 500 English reference queries with FastEmbed and with BGE-M3 and compare R@15, since a ~4-point gap on English-only retrieval would predict the observed QT/CLE gap.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section III-E states that document contexts are \"vectorized once per embedding model (FastEmbed for the Baseline/QT pipelines; LaBSE, multilingual E5, and BGE-M3 for the CLE pipeline)\" and stored in separate Pinecone indexes. Consequently, QT pipelines translate the Sinhala/Tamil query into English and then embed with FastEmbed, while CLE-BGE-M3 embeds both the source-language query and the English documents in BGE-M3's own space. The comparison between QT-GT and CLE-BGE-M3 therefore varies two independent factors simultaneously: (i) whether an explicit translation step is used and (ii) which embedding model performs retrieval. The paper's central claim — that cross-lingual embeddings outperform query translation, e.g., 96.2% vs 92.4% Recall@15 for Sinhala — could be explained entirely by replacing a lightweight English sentence embedder with a strong multilingual dense retriever, rather than by avoiding translation. This is not an external-consensus objection; it follows from the paper's own implementation. Furthermore, Section III-A's assertion that the architecture \"isolates retrieval-strategy effects from indexing effects\" is contradicted by III-E's model-specific document indexes. The conclusion acknowledges the lack of statistical testing and single-domain scope but does not acknowledge this encoder mismatch. Because QT-GT and CLE-BGE-M3 differ in both the translation step and the encoder, the 3.8–4.6 percentage-point gap cannot be attributed to the cross-lingual paradigm without a matched-encoder control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two cross-lingual information retrieval (CLIR) paradigms for retrieving English government documentation in response to Sinhala and Tamil queries: query translation (QT) using Google Translate, NLLB, and mBART50, and cross-lingual embeddings (CLE) using LaBSE, multilingual E5, and BGE-M3. A baseline with no cross-lingual adaptation is included. The experiments use a new benchmark of 500 Sinhala, Tamil, and English question-answer pairs built from 1,699 segmented contexts from Sri Lanka's Government Information Center. Retrieval is measured by Recall@k. The main empirical claim is that CLE with BGE-M3 achieves the highest Recall@15 (96.2% Sinhala-English, 95.6% Tamil-English), outperforming the best QT system, Google Translate (92.4% and 93.0%). The paper concludes that cross-lingual embeddings are a more effective and scalable foundation for cross-lingual RAG in low-resource government domains.","tokens_in":7428,"tokens_out":4018,"duration_ms":43241,"significance":"The paper addresses a practically important and under-studied problem: CLIR for Sinhala and Tamil in a real government-domain corpus. It contributes a new multilingual QA benchmark and compares six cross-lingual pipelines, including open-source and commercial components. The use of pretrained models without fine-tuning on the evaluation data is a strength, as it avoids circularity from fitting parameters to the benchmark. If the central comparison were made with matched retrieval encoders, the result that BGE-M3 CLE outperforms Google-Translate-based QT would be a useful practical finding for RAG systems in low-resource languages. However, the current evidence does not support the causal attribution of the performance gap to the translation-versus-alignment paradigm because the comparison is confounded by embedding model choice, and no statistical testing is provided.","major_comments":[{"comment":"The 'Sampled' column in Table I sums to 474, not 500 as stated in the text. Either one or more category counts are wrong or the total is misstated. Since the number of queries is central to all reported percentages, this inconsistency must be corrected.","section":"Table I"}],"minor_comments":[{"comment":"The paper states that the benchmark is 'publicly available,' but no repository, link, or license is provided. Please add an availability statement if the data is indeed public.","section":"Section IV-A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central comparison is potentially valuable but currently under-specified. The confound between translation strategy and embedding encoder is explicit in the methodology, and the missing significance tests and benchmark-construction details will require additional experiments or reanalysis. The citation of mBERT to the LaBSE paper and the Table I summation error suggest that a careful revision pass is needed. I would encourage the editor to request a revision rather than reject, because the underlying research question is timely and the benchmark could be a useful community resource if properly documented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives the community a genuinely new resource: a 500-question, human-verified English–Sinhala–Tamil QA benchmark built from Sri Lanka's Government Information Center, with retrieval-ready contexts. That alone justifies a look. The empirical sweep across three QT and three CLE models for two under-resourced languages is also more than most papers in this niche bother to do, and the authors are candid about the limits: no significance tests, single domain, no fine-tuning. The baseline failure (Recall@15 below 10%) is a useful sanity check. Credit where it's due: this is a solid, honest engineering comparison, not a fake result.\n\nThe soft spot is the one the stress-test note nails. The central claim—that cross-lingual embeddings beat query translation—is not actually supported by the numbers as run. Every QT pipeline translates the query into English and then embeds with FastEmbed; every CLE pipeline embeds the source-language query with the same multilingual model that indexed the documents. So QT–Google Translate versus CLE–BGE-M3 varies two things at once: the translation step and the encoder. The 3.8–4.6 point gap could just be BGE-M3 being a much stronger retriever than FastEmbed. The paper's own Section III-A says the architecture 'isolates retrieval-strategy effects from indexing effects,' but Section III-E describes model-specific document indexes. That contradiction is load-bearing. This isn't an external nitpick; it follows from their implementation. They needed a matched-encoder control—for example, BGE-M3 on the translated query as a QT variant—or a FastEmbed-only CLE variant, and they didn't run one.\n\nThe other issues are proportionally smaller. The benchmark is LLM-generated from the gold contexts, which can inflate absolute recall numbers, though the human verification and single-gold-passage design soften that. The lack of significance testing is acknowledged, but with 500 queries the difference between 96.2% and 92.4% is within the noise band they'd get from a simple binomial CI; they should report one. And despite claiming the benchmark is public, there's no link or repository in the text.\n\nWho is this for? Any practitioner building RAG for Sinhala or Tamil government services will want this paper for the benchmark and the relative ordering of off-the-shelf tools. As a research claim about CLIR paradigms, it's not there yet. I'd send it to review with a clear request: rerun the QT pipeline with a matched encoder, report uncertainty, and release the data. That is a serious referee's job, and the paper deserves it.","headline":"A useful new Sinhala–Tamil CLIR benchmark, but the headline QT-vs-CLE comparison is confounded by the use of different embedding models across pipelines.","tokens_in":7946,"tokens_out":1562,"would_cite":true,"duration_ms":18645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-lingual embeddings, led by BGE-M3, beat query translation for Sinhala- and Tamil-English retrieval, reaching about 96% Recall@15 and removing the translation step.","keywords":["Cross-Lingual Information Retrieval","Low-Resource Languages","Sinhala","Tamil","Query Translation","Cross-Lingual Embeddings","Retrieval-Augmented Generation","Recall@k"],"falsifier":"Embed Google-translated queries with the same BGE-M3 encoder used in the CLE pipeline (or another strong multilingual encoder) and re-run retrieval on the same index and query set; if that query-translation pipeline reaches or exceeds the reported 96.2%/95.6% $\\text{Recall@}15$, the paper's central claim that cross-lingual embeddings outperform translation by avoiding a translation step would be overturned.","tokens_in":6935,"feed_emoji":"🌐","tokens_out":9450,"duration_ms":83268,"temperature":0.7,"pith_summary":"The paper tries to establish that cross-lingual embedding models are a better bridge than query translation for retrieving English government documents from Sinhala and Tamil queries. On a human-verified benchmark of 500 queries built from 1,699 contexts in a Sri Lankan e-government corpus, BGE-M3 reaches $\\text{Recall@}15$ of 96.2% for Sinhala–English and 95.6% for Tamil–English, beating Google Translate's 92.4%/93.0% and needing no explicit translation step. The monolingual baseline that embeds non-English queries directly with an English-only model fails almost completely, below 10% recall. The result matters because it points to a scalable way to make retrieval-augmented generation work for low-resource languages without a fragile machine-translation component.","feed_headline":"BGE-M3 hits 96% recall for Sinhala and Tamil without translation","feed_subtitle":"On a Sri Lankan e-government benchmark, BGE-M3 beats Google Translate and lets retrieval skip translation entirely.","key_machinery":"The central machinery is a three-pipeline comparison in which the language gap between query and document is bridged either not at all, by translating the query, or by embedding the query in a shared multilingual space; all pipelines share the same English context index and cosine-similarity ranking, evaluated by $\\text{Recall@}k$. The load-bearing object is the multilingual bi-encoder embedding space: BGE-M3 is a multilingual embedding model trained with dense and hybrid retrieval objectives, and it places Sinhala/Tamil queries and English passages close together when they are semantically equivalent. That is what lets the CLE pipeline rank without translation, and the paper's argument is that this direct semantic alignment, rather than lexical translation, drives the recall advantage.","core_discovery":"In the paper's own terms, the discovery is that the cross-lingual embedding model BGE-M3 outperforms every query-translation pipeline for both Sinhala–English and Tamil–English retrieval on a real government-domain corpus, and does so while removing the translation step. Reported $\\text{Recall@}15$ numbers are 96.2% (Sinhala) and 95.6% (Tamil) for BGE-M3, versus 92.4%/93.0% for Google Translate, 87.0%/89.0% for NLLB, and 82.8%/85.2% for mBART50, with a monolingual baseline collapsing to 8.2%/4.2%. BGE-M3 also achieves the highest top-1 recall, near 60% for both languages, and the smallest language gap ($-0.6$ points), which the paper interprets as evidence that its multi-functional multilingual training generalizes across typologically distinct low-resource languages. The paper concludes that cross-lingual embeddings are a more effective and scalable foundation for cross-lingual RAG in low-resource settings than translation-based pipelines.","pith_inferences":["Because the QT pipeline always embeds translated queries with FastEmbed while CLE uses the stronger multilingual encoder, the headline comparison likely conflates paradigm with encoder quality; pairing Google Translate with a stronger English encoder might narrow or reverse the gap.","A natural extension is to test hybrid pipelines that translate the query and then embed with BGE-M3, or that merge QT and CLE rankings, since translation and embedding errors may be partly independent.","The single-gold-context benchmark may understate retrieval differences in open-ended RAG, where multiple relevant passages exist; a multi-gold or end-to-end generation evaluation could change the practical ranking of approaches.","If the Sinhala shortfall in multilingual E5 reflects pretraining corpus coverage, then per-language token counts should predict retrieval recall across models and languages; this can be tested without fine-tuning."],"forward_implications":["Government-facing RAG systems serving Sinhala and Tamil speakers can retrieve relevant English passages at around 95–96% recall without maintaining a machine-translation service, reducing latency and failure modes.","Google Translate remains a strong translation-based fallback, beating open-source NLLB by about 5 points at $\\text{Recall@}15$, so open-source deployments need to weigh that gap.","Using a monolingual English embedder directly on non-English queries is not viable for this domain, with recall below 10% at every depth.","BGE-M3's top-1 recall near 60% means the correct passage is usually first, which matters for RAG settings that only feed a few passages to the generator.","BGE-M3's near-zero Sinhala–Tamil gap suggests one multilingual model can serve both official languages of Sri Lanka without language-specific tuning."],"supporting_citations":[{"why":"Supplies BGE-M3, the multilingual dense/hybrid embedding model that is the top-performing CLE pipeline.","marker":"[10]"},{"why":"Supplies multilingual E5, the instruction-tuned CLE baseline that shows the largest Sinhala/Tamil gap.","marker":"[9]"},{"why":"Supplies LaBSE, the language-agnostic sentence embedding baseline for the CLE pipeline.","marker":"[7]"},{"why":"Supplies NLLB, the open-source translation model evaluated as a QT baseline.","marker":"[11]"},{"why":"Frames query translation vs document translation and the risk of translation error propagation into retrieval.","marker":"[5]"},{"why":"Supplies the RAG task formulation and the language-alignment assumption the paper targets.","marker":"[1]"},{"why":"Shows the existing multilingual retrieval benchmark that covers many languages but not Sinhala/Tamil, motivating the new benchmark.","marker":"[12]"},{"why":"Shows a multilingual passage-ranking dataset that likewise lacks these languages, reinforcing the gap the paper fills.","marker":"[13]"}],"fun_headline_variants":["BGE-M3 beats translation for Sinhala-Tamil e-gov retrieval","Skip translation: BGE-M3 hits 96% recall for Sinhala and Tamil","No translation needed: BGE-M3 tops Sinhala-Tamil retrieval","BGE-M3 outdoes Google Translate on Sinhala and Tamil queries","Cross-lingual embeddings win for Sinhala-Tamil e-gov search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the recall gap is due to the cross-lingual-embedding paradigm rather than to the fact that QT always uses a weaker English encoder while CLE uses BGE-M3, and that the percentage differences are meaningful without significance testing.","fun_headline_variants_meta":{"raw":{"variants":["BGE-M3 beats translation for Sinhala-Tamil e-gov retrieval","Skip translation: BGE-M3 hits 96% recall for Sinhala and Tamil","No translation needed: BGE-M3 tops Sinhala-Tamil retrieval","BGE-M3 outdoes Google Translate on Sinhala and Tamil queries","Cross-lingual embeddings win for Sinhala-Tamil e-gov search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001008,"raw_usage":{"total_tokens":4315,"prompt_tokens":1052,"completion_tokens":3263,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":3157}},"tokens_in":668,"tokens_out":3263,"duration_ms":22758,"temperature":1.0,"reasoning_tokens":3157,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:37:31.465764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Embed Google-translated queries with the same BGE-M3 encoder used in the CLE pipeline (or another strong multilingual encoder) and re-run retrieval on the same index and query set; if that query-translation pipeline reaches or exceeds the reported 96.2%/95.6% $\\text{Recall@}15$, the paper's central claim that cross-lingual embeddings outperform translation by avoiding a translation step would be overturned.","supporting_citations":[{"cited_title":"Document translation vs. query translation for cross-lingual information retrieval in the medical domain,","cited_arxiv_id":null,"evidence_quote":"Frames query translation vs document translation and the risk of translation error propagation into retrieval."}],"review_version":1}