Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO

G\"unter Neumann; Stalin Varanasi; Umer Butt

REVIEW 3 major objections 5 minor 12 references

Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that translating MS MARCO into Urdu with IndicTrans2 and fine-tuning mMARCO on that translation yields a retrieval model with MRR@10 of 0.248 and Recall@10 of 0.438, clearly above zero-shot and BM25 baselines.

desk verdict A useful dataset-and-baselines paper for Urdu IR: honest, open, and worth engaging with, but the headline numbers rest on an unvalidated machine-translated dev set and should be treated as provisional. read the letter →

arxiv 2412.12997 v3 pith:LIKAOSAL submitted 2024-12-17 cs.CL cs.AIcs.IR

Umer Butt , Stalin Varanasi , Günter Neumann This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords UrduinformationretrievalMSMARCOmachinetranslationlow-resourcelanguagesmT5BM25passagereranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper gives Urdu its first large-scale information-retrieval dataset by machine-translating the English MS MARCO passage-ranking corpus with IndicTrans2, then establishes retrieval baselines on it. A fine-tuned Urdu mT5-mMARCO reranker reaches MRR@10 of 0.248 and Recall@10 of 0.438, beating both a zero-shot mMARCO reranker and BM25. The authors argue that language-specific fine-tuning on machine-translated data is a practical route for low-resource-language retrieval, and they release the dataset, model, and code so others can build on the baselines.

What carries the argument

The load-bearing object is the Urdu mT5-mMARCO reranker: an mT5 sequence-to-sequence model, initialized from the multilingual mMARCO checkpoint, fine-tuned on 39 million machine-translated query-relevant-passage and query-nonrelevant-passage triples. At inference it scores each passage by the softmax probability of the Urdu word for "yes" versus "no," producing binary relevance decisions, and those scores rerank the top 1,000 passages retrieved by BM25. The other carrying component is the translation step itself: IndicTrans2 converts both queries and passages into Urdu while preserving the original relevance labels, which turns MS MARCO into a usable training and evaluation resource for Urdu.

What would settle it

Take a random sample of the 6,980 Urdu dev queries, have native Urdu speakers judge the relevance of the top passages returned by Urdu mT5-mMARCO, and compare those judgments with the machine-translated labels; strong disagreement would indicate that the reported MRR and Recall are inflated by translation artifacts, while agreement would confirm the scores.

Watch

Extended reading notes

Core claim

The central discovery is that translating an existing high-resource retrieval dataset and fine-tuning a multilingual reranker on the translation gives a substantial retrieval-quality boost for a low-resource language. On the Urdu dev set, BM25 achieves MRR@10 of 0.121 and Recall@10 of 0.247, zero-shot mMARCO achieves 0.204 and 0.408, and the fine-tuned Urdu mT5-mMARCO model achieves 0.248 and 0.438, with MAP@10 of 0.159 and NDCG@10 of 0.340. The authors take this as evidence that even imperfect machine-translated training data can adapt a multilingual model to a language it was never trained on, and that the resulting benchmark and baselines are a useful foundation for future Urdu IR work.

Load-bearing premise

The entire Urdu dataset is created by machine translation, so the scores depend on the assumption that IndicTrans2's translations preserve the original relevance labels well enough that retrieval metrics reflect genuine Urdu retrieval quality rather than adaptation to translation artifacts.

Editorial extensions

If this is right

Urdu IR now has a public benchmark and reproducible baselines, so future systems can be trained and evaluated in Urdu instead of relying on English or zero-shot multilingual transfer.
Fine-tuning on machine-translated data raises ranking quality above zero-shot transfer in this low-resource setting, suggesting a general recipe for languages that lack native IR datasets.
The two-stage pipeline of BM25 candidate retrieval plus a fine-tuned reranker transfers to Urdu without architectural changes, keeping the method simple and extensible.
Because dataset, model, and code are released, other South Asian languages with adequate translation models can follow the same path to establish their own first-pass benchmarks.
The fine-tuned model's gains over zero-shot mMARCO show that language-specific adaptation matters even when the underlying model is massively multilingual.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the evaluation set is itself machine-translated, the reported gains may partly measure how well the model adapts to translationese; a human-annotated Urdu test set would separate genuine language ability from translation-artifact adaptation.
Editorial inference: the same translate-then-finetune recipe may work for other low-resource languages, but its payoff probably depends on translation quality, so pairing it with post-editing or confidence-based filtering of translated training triples would be a natural next test.
Editorial inference: Urdu-specific tokenization and morphological normalization would likely raise the BM25 baseline itself, which would change the comparison point for all neural rerankers evaluated on this dataset.
Editorial inference: the framework's reliance on MS MARCO's labels means it cannot measure whether retrieval quality holds for natural Urdu queries outside the translated domain; a small, natively written Urdu query set would test that.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Plain take: this is a dataset-and-baselines paper, not a methods paper. The Urdu MS MARCO translation is new and fills a real gap; the baselines are honest and the resources are open. The central result — fine-tuning on translated Urdu data beats zero-shot — is credible but rests on an unvalidated machine-translated dev set, so the exact numbers should be treated as provisional.

What it does well: the authors translated the full MS MARCO corpus with IndicTrans2, fine-tuned the mMARCO mT5 reranker, and report BM25 and zero-shot comparisons. They give concrete training details (batch size, learning rate, hours on which GPU). They also flag in Section 5.2 that translation errors can break query-passage relevance, and they make data, model, and code public. That is the right way to ship a resource.

The soft spots are real but mostly minor. There are no confidence intervals or significance tests, so the gap between zero-shot MRR 0.204 and fine-tuned 0.248 could be noise. The zero-shot row omits MAP and NDCG; the abstract says 0.247 MRR and 0.439 Recall while Table 2 says 0.248 and 0.438. The bigger validity question is whether the dev set, translated from English, preserves relevance labels well enough. Since the model trains on the same translation style as the dev set, some of the apparent gain may be adaptation to translationese rather than real Urdu understanding. The stress-test note has this right. The authors acknowledge the issue but do not quantify it; Section 6 lists human verification and translation-quality analysis as future work. That is fair for a first dataset, but it means the headline numbers are not a final verdict on Urdu retrieval quality.

One thing the stress-test note does not mention: the BM25 baseline probably uses a generic tokenizer, which is a known weak spot for Urdu morphology. So the BM25 numbers are likely pessimistic, not just the neural numbers being optimistic.

Who this is for: IR researchers working on low-resource languages, especially South Asian languages. It is a useful starting resource and a reasonable baseline reference. It deserves a serious referee — the dataset itself is a contribution — but the review should push for a human-validated dev subset and error bars before the numbers are taken as definitive.

Recommendation: engage with it; cite it if you work in multilingual IR; send it to peer review with requested revisions.

Referee Report

3 major / 5 minor

Summary. The paper creates the first large-scale Urdu IR dataset by translating the English MS MARCO passage ranking dataset with IndicTrans2, and establishes retrieval baselines on it. The authors evaluate BM25, a zero-shot mMARCO re-ranker, and a fine-tuned Urdu mT5-mMARCO re-ranker on the Urdu-translated development set. They report that fine-tuning improves MRR@10 from 0.204 (zero-shot) and 0.121 (BM25) to 0.248, and Recall@10 from 0.408 and 0.247 to 0.438. The dataset, model, and code are released publicly.

Significance. If the reported results are reliable, the paper makes a useful resource contribution: it provides a large-scale Urdu IR dataset, reproducible baseline numbers, and an open-source fine-tuned model, which can support future work on Urdu and similar low-resource languages. The paper is transparent about the machine-translation provenance of the data and openly describes its limitations. However, the central empirical claim rests on an unvalidated assumption that the Urdu-translated development set preserves the English relevance labels; no human evaluation, label-preservation analysis, significance testing, or confidence intervals are provided. The contribution is therefore more of a promising first step than a fully verified benchmark, and this distinction needs to be made explicit in the paper.

major comments (3)

[Section 3.2 and Section 5.2] The dev set is entirely machine-translated with IndicTrans2, and relevance labels are inherited from English without human validation. Because the fine-tuned model is trained on the same translation style, its advantage over zero-shot mMARCO could partly reflect adaptation to translation artifacts rather than genuine Urdu retrieval ability. The paper itself acknowledges this risk (Section 5.2) and postpones a quantitative translation-quality analysis to future work (Section 6). To make the central claim load-bearing, I ask for a human-translated validation subset (e.g., 200-500 queries with human or independently adjudicated relevance labels) and a report of the same metrics on that subset, or at minimum a quantitative label-preservation analysis on a sample of the translated dev set.
[Table 2 and Section 4.3] The zero-shot mMARCO row is missing MAP@10 and NDCG@10, even though Section 4.3 states that these metrics are included in the evaluation. Since the paper's main comparison is between zero-shot and fine-tuned reranking, the table should report all four metrics for every row. Without the missing values and without any measure of uncertainty (e.g., bootstrap confidence intervals or paired significance tests over the 6,980 queries), the reported MRR gain from 0.204 to 0.248 and Recall gain from 0.408 to 0.438 cannot be assessed as statistically meaningful.
[Abstract and Table 2] The abstract reports MRR@10 of 0.247 and Recall@10 of 0.439, while Table 2 lists 0.248 and 0.438 for the same Urdu mT5-mMARCO model. The paper should reconcile these numbers and ensure the abstract matches the final table, since a discrepancy of 0.001 on the headline numbers undermines reader confidence in the precision of the reported results.

minor comments (5)

[Abstract] The phrase 'significant improvements' is used in the abstract and Section 5.1 without any statistical testing; consider replacing it with 'improvements' or adding the appropriate significance evidence, as noted in the major comments.
[Section 2 and Section 4.2] The model name is spelled inconsistently as mMARCO, mMarco, and mMarco; please standardize to a single spelling (mMARCO) throughout.
[Table 1] Table 1 is difficult to read in the current format: the highlighted omission of the word 'native' is mentioned in Section 5.2, but the table itself is not self-contained and the highlighting is not described in the caption. Please add a note about the coloring and ensure the Urdu examples are clearly aligned with their English counterparts.
[Section 2] The sentence 'MIRACL addresses this by providing a diverse set of 18 languages' is slightly misleading because MIRACL is primarily a retrieval dataset rather than an initiative that 'addresses' low-resource IR in the same sense as this paper; consider rephrasing to describe what MIRACL actually offers.
[References] Reference [11] is cited as 'Xue, L.: mt5: A massively multilingual pre-trained text-to-text transformer' but the arXiv identifier 2010.11934 corresponds to the mT5 paper; the citation should include the full author list and the correct title format.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the fine-tuned model's gains are empirical outcomes on an out-of-sample dev set, and the acknowledged translation-quality issue is a validity concern, not a circularity concern.

full rationale

The paper's derivation chain is: translate the English MS MARCO training and dev sets into Urdu with IndicTrans2; fine-tune the mMARCO model on Urdu-translated triples; and evaluate on the Urdu-translated dev set with relevance labels inherited from the original English MS MARCO. The reported MRR@10 and Recall@10 improvements are measured system outputs, not quantities defined in terms of fitted parameters or evaluation labels. No parameter is fitted to the dev set, and the dev set is not used in training. The baselines, BM25 and zero-shot mMARCO, are external to the paper, and the fine-tuned model is compared on held-out dev queries. The paper contains no load-bearing self-citations, no imported uniqueness theorem, and no renaming of a known result. The main caveat, stated in Section 5.2, is that machine-translated dev data may misalign query-passage semantics, such as the word 'native' being omitted in a passage, so metrics may partly reflect adaptation to translationese rather than genuine Urdu retrieval quality. That is a validity threat to the evaluation, not a circularity in the derivation, and the paper explicitly acknowledges this limitation and lists manual verification as future work. Therefore the circularity score is 0.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

No new theoretical entities or fitted constants are introduced. The resource creation relies on the quality of IndicTrans2 translations and on the transferability of English relevance labels. Hyperparameters are fixed following the mMARCO setup.

free parameters (1)

Fine-tuning hyperparameters (learning rate, effective batch size) = learning rate 0.001, effective batch size 128
Chosen by hand following the original mMARCO setup (Section 4.2), not tuned on the Urdu dev set. The exact values are unlikely to change the central conclusion but are parameters the authors fixed.

assumptions (2)

domain assumption IndicTrans2 translation preserves enough semantic fidelity to train an effective retrieval model.
The Urdu dataset is created solely by machine translation of the English MS MARCO using IndicTrans2; if translation quality is insufficient, the training and evaluation data would be noisy and the baselines would be biased. Authors acknowledge this in Section 5.2.
domain assumption Relevance labels in English MS MARCO remain valid after translation to Urdu.
The paper uses the original English relevance judgments as ground truth for the Urdu-translated queries and passages. Errors in translation could break query-passage semantic alignment, invalidating the evaluation. This is assumed throughout the dataset creation and evaluation (Sections 3.2 and 4.3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO." pith.science (2026). https://pith.science/paper/LIKAOSAL

@misc{pith2026241212997,
  author       = {Pith},
  title        = {Pith review of: Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LIKAOSAL}},
  note         = {Machine review of arXiv:2412.12997}
}

read the original abstract

As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. This paper introduces the first large-scale Urdu IR dataset, created by translating the MS MARCO dataset through machine translation. We establish baseline results through zero-shot learning for IR in Urdu and subsequently apply the mMARCO multilingual IR methodology to this newly translated dataset. Our findings demonstrate that the fine-tuned model (Urdu-mT5-mMARCO) achieves a Mean Reciprocal Rank (MRR@10) of 0.247 and a Recall@10 of 0.439, representing significant improvements over zero-shot results and showing the potential for expanding IR access for Urdu speakers. By bridging access gaps for speakers of low-resource languages, this work not only advances multilingual IR research but also emphasizes the ethical and societal importance of inclusive IR technologies. This work provides valuable insights into the challenges and solutions for improving language representation and lays the groundwork for future research, especially in South Asian languages, which can benefit from the adaptable methods used in this study.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

[1]

In: Proceedings of the 15th Annual Meeting of the Forum for Informa- tion Retrieval Evaluation

Adeyemi, M., Oladipo, A., Zhang, X., Alfonso-Hermelo, D., Rezagholizadeh, M., Chen, B., Lin, J.: Ciral at fire 2023: Cross-lingual information retrieval for african languages. In: Proceedings of the 15th Annual Meeting of the Forum for Informa- tion Retrieval Evaluation. pp. 4–6 (2023)

work page 2023
[2]

Generalists vs. Specialists: Evaluating Large Language Models for Urdu

Arif,S.,Azeemi,A.H.,Raza,A.A.,Athar,A.:Generalistsvs.specialists:Evaluating large language models for urdu. arXiv preprint arXiv:2407.04459 (2024)

work page Pith review arXiv 2024
[3]

arXiv preprint arXiv:1611.09268 (2016)

Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., Mc- Namara, A., Mitra, B., Nguyen, T., et al.: Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016)

arXiv 2016
[4]

arXiv preprint arXiv:2108.13897 (2021)

Bonifacio, L., Jeronymo, V., Abonizio, H.Q., Campiotti, I., Fadaee, M., Lotufo, R., Nogueira, R.: mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv preprint arXiv:2108.13897 (2021)

arXiv 2021
[5]

arXiv preprint arXiv:2305.16307 (2023)

Gala, J., Chitale, P.A., AK, R., Gumma, V., Doddapaneni, S., Kumar, A., Nawale, J., Sujatha, A., Puduppully, R., Raghavan, V., et al.: Indictrans2: Towards high- quality and accessible machine translation models for all 22 scheduled indian lan- guages. arXiv preprint arXiv:2305.16307 (2023)

arXiv 2023
[6]

In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval

Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via con- textualized late interaction over bert. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 39–48 (2020)

2020
[7]

arXiv preprint arXiv:2212.10528 (2022)

Lu, J., Hall, K., Ma, J., Ni, J.: Hyrr: Hybrid infused reranking for passage retrieval. arXiv preprint arXiv:2212.10528 (2022)

arXiv 2022
[8]

In: European Conference on Information Retrieval

Nair, S., Yang, E., Lawrie, D., Duh, K., McNamee, P., Murray, K., Mayfield, J., Oard, D.W.: Transfer learning approaches for building cross-language dense re- trieval models. In: European Conference on Information Retrieval. pp. 382–396. Springer (2022)

work page 2022

Show all 12 references

[9]

Nist Special Publication Sp pp

Robertson, S.E., Walker, S., Beaulieu, M., Gatford, M., Payne, A.: Okapi at trec-4. Nist Special Publication Sp pp. 73–96 (1996)

1996
[10]

Language Resources and Evaluation (58), 713–755 (2023)

Tiedemann, J., Aulamo, M., Bakshandaeva, D., Boggia, M., Grönroos, S.A., Niem- inen, T., Raganato A., Scherrer, Y., Vazquez, R., Virpioja, S.: Democratizing neu- ral machine translation with OPUS-MT. Language Resources and Evaluation (58), 713–755 (2023). https://doi.org/10.10...

2023 doi
[11]

arXiv preprint arXiv:2010.11934 (2020)

Xue, L.: mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 (2020)

2020 arXiv
[12]

Transactions of the Association for Computational Linguistics 11, 1114–1131 (2023)

Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., Lin, J.: Miracl: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics 11, 1114–1131 (2023)

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

In: Proceedings of the 15th Annual Meeting of the Forum for Informa- tion Retrieval Evaluation

Adeyemi, M., Oladipo, A., Zhang, X., Alfonso-Hermelo, D., Rezagholizadeh, M., Chen, B., Lin, J.: Ciral at fire 2023: Cross-lingual information retrieval for african languages. In: Proceedings of the 15th Annual Meeting of the Forum for Informa- tion Retrieval Evaluation. pp. 4–6 (2023)

work page 2023

[2] [2]

Generalists vs. Specialists: Evaluating Large Language Models for Urdu

Arif,S.,Azeemi,A.H.,Raza,A.A.,Athar,A.:Generalistsvs.specialists:Evaluating large language models for urdu. arXiv preprint arXiv:2407.04459 (2024)

work page Pith review arXiv 2024

[3] [3]

arXiv preprint arXiv:1611.09268 (2016)

Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., Mc- Namara, A., Mitra, B., Nguyen, T., et al.: Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016)

arXiv 2016

[4] [4]

arXiv preprint arXiv:2108.13897 (2021)

Bonifacio, L., Jeronymo, V., Abonizio, H.Q., Campiotti, I., Fadaee, M., Lotufo, R., Nogueira, R.: mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv preprint arXiv:2108.13897 (2021)

arXiv 2021

[5] [5]

arXiv preprint arXiv:2305.16307 (2023)

Gala, J., Chitale, P.A., AK, R., Gumma, V., Doddapaneni, S., Kumar, A., Nawale, J., Sujatha, A., Puduppully, R., Raghavan, V., et al.: Indictrans2: Towards high- quality and accessible machine translation models for all 22 scheduled indian lan- guages. arXiv preprint arXiv:2305.16307 (2023)

arXiv 2023

[6] [6]

In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval

Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via con- textualized late interaction over bert. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 39–48 (2020)

2020

[7] [7]

arXiv preprint arXiv:2212.10528 (2022)

Lu, J., Hall, K., Ma, J., Ni, J.: Hyrr: Hybrid infused reranking for passage retrieval. arXiv preprint arXiv:2212.10528 (2022)

arXiv 2022

[8] [8]

In: European Conference on Information Retrieval

Nair, S., Yang, E., Lawrie, D., Duh, K., McNamee, P., Murray, K., Mayfield, J., Oard, D.W.: Transfer learning approaches for building cross-language dense re- trieval models. In: European Conference on Information Retrieval. pp. 382–396. Springer (2022)

work page 2022