Query Translation vs. Cross-Lingual Embeddings for Sinhala-Tamil E-Government Information Retrieval

Dharshi Balasubramaniyam; Tiroshan Madushanka

REVIEW 1 major objections 1 minor 14 references

Query Translation vs. Cross-Lingual Embeddings for Sinhala-Tamil E-Government Information Retrieval

T0 review · 1 major / 1 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cross-lingual embeddings, led by BGE-M3, beat query translation for Sinhala- and Tamil-English retrieval, reaching about 96% Recall@15 and removing the translation step.

desk verdict A useful new Sinhala–Tamil CLIR benchmark, but the headline QT-vs-CLE comparison is confounded by the use of different embedding models across pipelines. read the letter →

arxiv 2608.12820 v1 pith:XPTC22KA submitted 2026-08-13 cs.IR

Dharshi Balasubramaniyam , Tiroshan Madushanka This is my paper · ORCID

classification cs.IR

keywords Cross-LingualInformationRetrievalLow-ResourceLanguagesSinhalaTamilQueryTranslationEmbeddingsRetrieval-AugmentedGenerationRecall@k

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that cross-lingual embedding models are a better bridge than query translation for retrieving English government documents from Sinhala and Tamil queries. On a human-verified benchmark of 500 queries built from 1,699 contexts in a Sri Lankan e-government corpus, BGE-M3 reaches $\text{Recall@}15$ of 96.2% for Sinhala–English and 95.6% for Tamil–English, beating Google Translate's 92.4%/93.0% and needing no explicit translation step. The monolingual baseline that embeds non-English queries directly with an English-only model fails almost completely, below 10% recall. The result matters because it points to a scalable way to make retrieval-augmented generation work for low-resource languages without a fragile machine-translation component.

What carries the argument

The central machinery is a three-pipeline comparison in which the language gap between query and document is bridged either not at all, by translating the query, or by embedding the query in a shared multilingual space; all pipelines share the same English context index and cosine-similarity ranking, evaluated by $\text{Recall@}k$. The load-bearing object is the multilingual bi-encoder embedding space: BGE-M3 is a multilingual embedding model trained with dense and hybrid retrieval objectives, and it places Sinhala/Tamil queries and English passages close together when they are semantically equivalent. That is what lets the CLE pipeline rank without translation, and the paper's argument is that this direct semantic alignment, rather than lexical translation, drives the recall advantage.

What would settle it

Embed Google-translated queries with the same BGE-M3 encoder used in the CLE pipeline (or another strong multilingual encoder) and re-run retrieval on the same index and query set; if that query-translation pipeline reaches or exceeds the reported 96.2%/95.6% $\text{Recall@}15$, the paper's central claim that cross-lingual embeddings outperform translation by avoiding a translation step would be overturned.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that the cross-lingual embedding model BGE-M3 outperforms every query-translation pipeline for both Sinhala–English and Tamil–English retrieval on a real government-domain corpus, and does so while removing the translation step. Reported $\text{Recall@}15$ numbers are 96.2% (Sinhala) and 95.6% (Tamil) for BGE-M3, versus 92.4%/93.0% for Google Translate, 87.0%/89.0% for NLLB, and 82.8%/85.2% for mBART50, with a monolingual baseline collapsing to 8.2%/4.2%. BGE-M3 also achieves the highest top-1 recall, near 60% for both languages, and the smallest language gap ($-0.6$ points), which the paper interprets as evidence that its multi-functional multilingual training generalizes across typologically distinct low-resource languages. The paper concludes that cross-lingual embeddings are a more effective and scalable foundation for cross-lingual RAG in low-resource settings than translation-based pipelines.

Load-bearing premise

The load-bearing premise is that the recall gap is due to the cross-lingual-embedding paradigm rather than to the fact that QT always uses a weaker English encoder while CLE uses BGE-M3, and that the percentage differences are meaningful without significance testing.

Editorial extensions

If this is right

Government-facing RAG systems serving Sinhala and Tamil speakers can retrieve relevant English passages at around 95–96% recall without maintaining a machine-translation service, reducing latency and failure modes.
Google Translate remains a strong translation-based fallback, beating open-source NLLB by about 5 points at $\text{Recall@}15$, so open-source deployments need to weigh that gap.
Using a monolingual English embedder directly on non-English queries is not viable for this domain, with recall below 10% at every depth.
BGE-M3's top-1 recall near 60% means the correct passage is usually first, which matters for RAG settings that only feed a few passages to the generator.
BGE-M3's near-zero Sinhala–Tamil gap suggests one multilingual model can serve both official languages of Sri Lanka without language-specific tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the QT pipeline always embeds translated queries with FastEmbed while CLE uses the stronger multilingual encoder, the headline comparison likely conflates paradigm with encoder quality; pairing Google Translate with a stronger English encoder might narrow or reverse the gap.
A natural extension is to test hybrid pipelines that translate the query and then embed with BGE-M3, or that merge QT and CLE rankings, since translation and embedding errors may be partly independent.
The single-gold-context benchmark may understate retrieval differences in open-ended RAG, where multiple relevant passages exist; a multi-gold or end-to-end generation evaluation could change the practical ranking of approaches.
If the Sinhala shortfall in multilingual E5 reflects pretraining corpus coverage, then per-language token counts should predict retrieval recall across models and languages; this can be tested without fine-tuning.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives the community a genuinely new resource: a 500-question, human-verified English–Sinhala–Tamil QA benchmark built from Sri Lanka's Government Information Center, with retrieval-ready contexts. That alone justifies a look. The empirical sweep across three QT and three CLE models for two under-resourced languages is also more than most papers in this niche bother to do, and the authors are candid about the limits: no significance tests, single domain, no fine-tuning. The baseline failure (Recall@15 below 10%) is a useful sanity check. Credit where it's due: this is a solid, honest engineering comparison, not a fake result.

The soft spot is the one the stress-test note nails. The central claim—that cross-lingual embeddings beat query translation—is not actually supported by the numbers as run. Every QT pipeline translates the query into English and then embeds with FastEmbed; every CLE pipeline embeds the source-language query with the same multilingual model that indexed the documents. So QT–Google Translate versus CLE–BGE-M3 varies two things at once: the translation step and the encoder. The 3.8–4.6 point gap could just be BGE-M3 being a much stronger retriever than FastEmbed. The paper's own Section III-A says the architecture 'isolates retrieval-strategy effects from indexing effects,' but Section III-E describes model-specific document indexes. That contradiction is load-bearing. This isn't an external nitpick; it follows from their implementation. They needed a matched-encoder control—for example, BGE-M3 on the translated query as a QT variant—or a FastEmbed-only CLE variant, and they didn't run one.

The other issues are proportionally smaller. The benchmark is LLM-generated from the gold contexts, which can inflate absolute recall numbers, though the human verification and single-gold-passage design soften that. The lack of significance testing is acknowledged, but with 500 queries the difference between 96.2% and 92.4% is within the noise band they'd get from a simple binomial CI; they should report one. And despite claiming the benchmark is public, there's no link or repository in the text.

Who is this for? Any practitioner building RAG for Sinhala or Tamil government services will want this paper for the benchmark and the relative ordering of off-the-shelf tools. As a research claim about CLIR paradigms, it's not there yet. I'd send it to review with a clear request: rerun the QT pipeline with a matched encoder, report uncertainty, and release the data. That is a serious referee's job, and the paper deserves it.

Referee Report

1 major / 1 minor

Summary. The paper compares two cross-lingual information retrieval (CLIR) paradigms for retrieving English government documentation in response to Sinhala and Tamil queries: query translation (QT) using Google Translate, NLLB, and mBART50, and cross-lingual embeddings (CLE) using LaBSE, multilingual E5, and BGE-M3. A baseline with no cross-lingual adaptation is included. The experiments use a new benchmark of 500 Sinhala, Tamil, and English question-answer pairs built from 1,699 segmented contexts from Sri Lanka's Government Information Center. Retrieval is measured by Recall@k. The main empirical claim is that CLE with BGE-M3 achieves the highest Recall@15 (96.2% Sinhala-English, 95.6% Tamil-English), outperforming the best QT system, Google Translate (92.4% and 93.0%). The paper concludes that cross-lingual embeddings are a more effective and scalable foundation for cross-lingual RAG in low-resource government domains.

Significance. The paper addresses a practically important and under-studied problem: CLIR for Sinhala and Tamil in a real government-domain corpus. It contributes a new multilingual QA benchmark and compares six cross-lingual pipelines, including open-source and commercial components. The use of pretrained models without fine-tuning on the evaluation data is a strength, as it avoids circularity from fitting parameters to the benchmark. If the central comparison were made with matched retrieval encoders, the result that BGE-M3 CLE outperforms Google-Translate-based QT would be a useful practical finding for RAG systems in low-resource languages. However, the current evidence does not support the causal attribution of the performance gap to the translation-versus-alignment paradigm because the comparison is confounded by embedding model choice, and no statistical testing is provided.

major comments (1)

[Table I] The 'Sampled' column in Table I sums to 474, not 500 as stated in the text. Either one or more category counts are wrong or the total is misstated. Since the number of queries is central to all reported percentages, this inconsistency must be corrected.

minor comments (1)

[Section IV-A.1] The paper states that the benchmark is 'publicly available,' but no repository, link, or license is provided. Please add an availability statement if the data is indeed public.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical QT-vs-CLE comparison is measured directly, and the noted encoder confound is a benchmark-design issue, not a circular derivation.

full rationale

The paper's central claim is an empirical ranking of retrieval pipelines measured by Recall@k on a fixed query set. No parameter is fitted to the evaluation data: all embedding and translation models are used in pre-trained form, and the paper explicitly notes that none were fine-tuned. No load-bearing result is justified by self-citation; the references to prior models provide external context rather than circular support. The Recall@k values are direct measurements, not quantities derived from their own inputs by construction. The strongest concern raised by a reader is that QT pipelines embed translated queries with FastEmbed while CLE-BGE-M3 embeds queries and documents with BGE-M3, so the comparison conflates the translation paradigm with encoder choice. That is a genuine confound affecting whether the gap is attributable to cross-lingual embeddings, but it is not circularity: the reported scores are not statistically forced, renamed fits, or definitions in disguise. Likewise, the internally constructed LLM-generated benchmark may make retrieval artificially easy, but that is an external-validity limitation, not a circular derivation chain. The paper also candidly acknowledges the lack of statistical testing and the single-domain scope. Accordingly, no circular step meets the evidentiary bar, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters or new theoretical entities. Its claims rest on the validity of the generated benchmark and on the comparability of the QT and CLE pipelines, the latter being an unstated and questionable assumption.

assumptions (4)

domain assumption Each query has exactly one gold context, the context from which its QA pair was generated.
The evaluation assumes the generated question is answerable only by its source context; if multiple contexts contain the answer, Recall@k is understated or the gold mapping is ambiguous. See Section IV.A.1.
domain assumption The human verification of the LLM-generated QA pairs ensures queries are linguistically correct and uniquely mapped to the source context.
No details on the verification process, number of annotators, or agreement are given; the validity of the gold labels depends on this. See Section IV.A.1.
domain assumption Cosine similarity in the shared embedding space is a valid relevance signal for the retrieval task.
All pipelines rank by cosine similarity; the paper does not calibrate or threshold this score. See Section III.E.
ad hoc to paper The use of FastEmbed as the retrieval encoder in QT pipelines does not disadvantage QT relative to CLE.
This is the central confound: QT uses a lightweight English embedder while CLE uses each multilingual model as its own encoder. The assumption is unspoken and load-bearing. See Section III.C and III.E.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Query Translation vs. Cross-Lingual Embeddings for Sinhala-Tamil E-Government Information Retrieval." pith.science (2026). https://pith.science/paper/XPTC22KA

@misc{pith2026260812820,
  author       = {Pith},
  title        = {Pith review of: Query Translation vs. Cross-Lingual Embeddings for Sinhala-Tamil E-Government Information Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPTC22KA}},
  note         = {Machine review of arXiv:2608.12820}
}

read the original abstract

This paper presents a comparative evaluation of cross-lingual information retrieval (CLIR) methods for retrieving English government information using Sinhala and Tamil queries. Two CLIR paradigms are investigated: Query Translation (QT), employing Google Translate, NLLB, and mBART50, and Cross-Lingual Embeddings (CLE), using LaBSE, multilingual E5, and BGE-M3, with monolingual English retrieval as the baseline. Experiments are conducted on a human-verified benchmark comprising 500 Sinhala, Tamil, and English question-answer pairs derived from 1,699 segmented contexts from Sri Lanka's Government Information Center (GIC). Retrieval performance is evaluated using Recall@k (k = 1, 3, 5, 10, 15). Monolingual retrieval performs poorly (Recall@15 <10%), whereas all CLIR approaches substantially improve retrieval accuracy. Among them, BGE-M3 achieves the highest Recall@15, reaching 96.2% for Sinhala-English and 95.6% for Tamil-English, outperforming the best QT approach (Google Translate: 92.4% and 93.0%) while avoiding translation overhead. These results demonstrate that multilingual embedding models provide a more effective and scalable solution for cross-lingual retrieval-augmented generation (RAG) in low-resource government domains.

Figures

Figures reproduced from arXiv: 2608.12820 by the authors.

**Figure 1.** Proposed three-pipeline retrieval architecture. All pipelines share the same indexed English knowledge base and return the top-15 ranked contexts by [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Recall@15 across all evaluated retrieval approaches for Sinhala– [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Recall@k trend for the strongest CLE and QT pipelines at increasing [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 5 canonical work pages

[1]

Retrieval-augmented generation for knowledge-intensive NLP tasks,

P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” arXiv preprint arXiv:2005.11401, 2020

arXiv 2005
[2]

Retrieval-augmented generation for large language models: A survey,

Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,”arXiv preprint arXiv:2312.10997, 2023

arXiv 2023
[3]

Cross-lingual training of dense retrievers for document retrieval,

P. Shi, R. Zhang, H. Bai, and J. Lin, “Cross-lingual training of dense retrievers for document retrieval,” inProceedings of the 1st Workshop on Multilingual Representation Learning (MRL), 2021

work page 2021
[4]

Bridging language gaps: Advances in cross-lingual information retrieval with multilingual LLMs,

R. Goworek, O. Macmillan-Scott, and E. B. ¨Ozyi˘git, “Bridging language gaps: Advances in cross-lingual information retrieval with multilingual LLMs,”arXiv preprint arXiv:2510.00908, 2025

work page arXiv 2025
[5]

Document translation vs. query translation for cross-lingual information retrieval in the medical domain,

S. Saleh and P. Pecina, “Document translation vs. query translation for cross-lingual information retrieval in the medical domain,” inProceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 6849–6860

work page 2020
[6]

Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval

Z. Huang, H. Bonab, S. M. Sarwar, R. Rahimi, and J. Allan, “Mixed attention transformer for leveraging word-level knowledge to neural cross-lingual information retrieval,”arXiv preprint arXiv:2109.02789, 2021

work page Pith review arXiv 2021
[7]

Language- agnostic BERT sentence embedding,

F. Feng, Y . Yang, D. Cer, N. Arivazhagan, and W. Wang, “Language- agnostic BERT sentence embedding,”arXiv preprint arXiv:2007.01852, 2020

arXiv 2007
[8]

Unsu- pervised cross-lingual representation learning at scale,

A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzm´an, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsu- pervised cross-lingual representation learning at scale,”arXiv preprint arXiv:1911.02116, 2019

arXiv 1911

Show all 14 references

[9]

Multilingual E5 text embeddings: A technical report,

L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei, “Multilingual E5 text embeddings: A technical report,”arXiv preprint arXiv:2402.05672, 2024

2024 arXiv
[10]

BGE M3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,

J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “BGE M3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,”arXiv preprint arXiv:2402.03216, 2024

2024 arXiv
[11]

No language left behind: Scaling human-centered machine translation,

NLLB Team, M. R. Costa-juss `a, J. Cross, O. C ¸ elebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Bar- rault, G. M. Gonzalez, P. Hansanti, and J. Wang, “No language left behind: Sc...

2022 arXiv
[12]

MIRACL: A multilingual retrieval dataset covering 18 diverse languages,

X. Zhang, N. Thakur, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, M. Rezagholizadeh, and J. Lin, “MIRACL: A multilingual retrieval dataset covering 18 diverse languages,”Transactions of the Association for Computational Linguistics, vol. 11, pp. 1114–1131, 2023

2023
[13]

mMARCO: A multilingual version of the MS MARCO passage ranking dataset,

L. H. Bonifacio, H. Abonizio, M. Fadaee, and R. Nogueira, “mMARCO: A multilingual version of the MS MARCO passage ranking dataset,” arXiv preprint arXiv:2108.13897, 2021

2021 arXiv
[14]

Mind the gap: Cross-lingual information retrieval with hierarchical knowledge enhancement,

F. Zhang, Z. Zhang, X. Ao, D. Gao, F. Zhuang, Y . Wei, and Q. He, “Mind the gap: Cross-lingual information retrieval with hierarchical knowledge enhancement,”arXiv preprint arXiv:2112.13510, 2021

2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Retrieval-augmented generation for knowledge-intensive NLP tasks,

P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” arXiv preprint arXiv:2005.11401, 2020

arXiv 2005

[2] [2]

Retrieval-augmented generation for large language models: A survey,

Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,”arXiv preprint arXiv:2312.10997, 2023

arXiv 2023

[3] [3]

Cross-lingual training of dense retrievers for document retrieval,

P. Shi, R. Zhang, H. Bai, and J. Lin, “Cross-lingual training of dense retrievers for document retrieval,” inProceedings of the 1st Workshop on Multilingual Representation Learning (MRL), 2021

work page 2021

[4] [4]

Bridging language gaps: Advances in cross-lingual information retrieval with multilingual LLMs,

R. Goworek, O. Macmillan-Scott, and E. B. ¨Ozyi˘git, “Bridging language gaps: Advances in cross-lingual information retrieval with multilingual LLMs,”arXiv preprint arXiv:2510.00908, 2025

work page arXiv 2025

[5] [5]

Document translation vs. query translation for cross-lingual information retrieval in the medical domain,

S. Saleh and P. Pecina, “Document translation vs. query translation for cross-lingual information retrieval in the medical domain,” inProceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 6849–6860

work page 2020

[6] [6]

Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval

Z. Huang, H. Bonab, S. M. Sarwar, R. Rahimi, and J. Allan, “Mixed attention transformer for leveraging word-level knowledge to neural cross-lingual information retrieval,”arXiv preprint arXiv:2109.02789, 2021

work page Pith review arXiv 2021

[7] [7]

Language- agnostic BERT sentence embedding,

F. Feng, Y . Yang, D. Cer, N. Arivazhagan, and W. Wang, “Language- agnostic BERT sentence embedding,”arXiv preprint arXiv:2007.01852, 2020

arXiv 2007

[8] [8]

Unsu- pervised cross-lingual representation learning at scale,

A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzm´an, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsu- pervised cross-lingual representation learning at scale,”arXiv preprint arXiv:1911.02116, 2019

arXiv 1911