Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Data Augmentation With Back translation for Low Resource languages: A case of English and Luganda

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read Selective back translation lifts English–Luganda BLEU by more than 10 points, the paper reports.

desk verdict Useful data effort and a plausible internal result, but the +10 BLEU claim over prior work is not established and the selection step risks overfitting to the test set. read the letter →

arxiv 2505.02463 v1 pith:53W7IJKY submitted 2025-05-05 cs.CL

classification cs.CL
keywords NeuralMachineTranslationBackLowResourceLanguagesEnglish-LugandaLanguagePairDataAugmentationQualityMetricsIterativeIncremental
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that back translation—translating monolingual text with a preliminary model to create synthetic parallel sentences—can give a large boost to neural machine translation for a low-resource language pair, English and Luganda. The authors build bilingual and monolingual datasets from public and web-crawled sources, then run iterative and incremental back translation with a dataset-selection step they call OurBT. They report final BLEU scores of 40.25 for English-to-Luganda and 44.25 for Luganda-to-English on their newly built 'newtest' set, more than 10 BLEU points above earlier back-translation results for this pair. If the comparison holds, it would mean a practical recipe for improving low-resource translation using small, fragmented monolingual corpora, and the paper releases the datasets to make the result reproducible. The claim depends on the new test set being comparable to earlier evaluations.

What carries the argument

The central mechanism is the paper's 'OurBT' algorithm: starting from a bilingual baseline model, it translates each small monolingual dataset in both directions, retrains a model on each resulting synthetic set, ranks the datasets by the BLEU score of the retrained models, and then retrains the final model on synthetic translations of the top-ranked monolingual sets. Iterating this cycle implements iterative back translation; adding monolingual data in growing portions implements incremental back translation. The mechanism acts as a selection filter: it scores fragmented monolingual corpora by how much they help the translation task, and the paper finds that the best-scoring sets are those with contextual similarity to the training data.

What would settle it

Run the earlier back-translation system [3] on this paper's 'newtest' sentences and compute its BLEU; if it scores within 10 BLEU points of 40.25 (Eng2Lug) or 44.25 (Lug2Eng), the central comparative claim is not supported. The mirror-image check is to run this paper's final model on the earlier work's test set and see whether the gap survives.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that a back-translation routine that selects the best-performing small monolingual datasets produces large gains for English–Luganda: BLEU rises from 29.67 to 40.25 for English-to-Luganda and from 32.92 to 44.25 for Luganda-to-English on the newtest split, more than 10 points above the earlier back-translation scores of 26.7 and 33.2 [3]. The paper also claims that the improvement is not an artifact of BLEU alone, since ChrF2 rises and TER falls in both directions, and sample translations align with reference translations better than a commercial online translator. It attributes the gains to selecting datasets whose context matches the training data rather than to simply adding more monolingual text.

Load-bearing premise

The headline claim assumes that BLEU scores on the newly built 'newtest' set can be compared directly with earlier reported BLEU scores obtained on different test sets; if the test sets differ, the claimed 10-point lead is not established.

Editorial extensions

If this is right

  • English–Luganda translation can reach BLEU scores of 40.25 and 44.25 on the newtest split using back translation over small, selected monolingual datasets, a level that would make the pair substantially more usable.
  • Dataset selection matters: applying standard back translation to the best dataset combination gives only +2.49 BLEU for Eng2Lug and +0.45 for Lug2Eng, while the selection-then-retrain routine gives +6.27 and +7.05 before iteration.
  • Repeating the selection-retrain cycle improves scores in each of three reported iterations, suggesting the routine can be run until convergence rather than fixed at a single pass.
  • The publicly released bilingual and monolingual datasets give other researchers a common resource for English–Luganda NMT, addressing the previous lack of public data for this pair.
  • Evaluation with ChrF2 and TER gives a fuller picture for a morphologically rich language: TER falls from 69.7 to 65.4 for Eng2Lug and from 64.7 to 60.4 for Lug2Eng, consistent with the BLEU gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 10-point margin over prior work is only meaningful if the test sets are comparable; a head-to-head evaluation on a shared held-out set could confirm or shrink the gap.
  • The fact that context-matching datasets win selection suggests back translation here is doing implicit domain adaptation, so a cheaper selector could rank monolingual datasets by domain similarity without retraining a full model per candidate.
  • The recipe—small monolingual corpora from varied sources, scored one at a time—should transfer to other Bantu and low-resource languages with fragmented data; testing it on Swahili, Acholi, or Runyankore would show whether the selection step generalises.
  • The large gap between the default validation scores and the newtest scores (for example, 53.77 versus 29.67 for Eng2Lug) shows how much test-set choice affects apparent quality; future benchmarks for this language pair should settle on a shared public test split to make comparisons meaningful.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies back translation (BT) for English–Luganda neural machine translation. It assembles bilingual data from four sources and monolingual data for both languages, including web-crawled news text, trains Transformer models, and applies standard, incremental, iterative, and a proposed 'OurBT' variant. OurBT (Algorithm 1) selects the best monolingual dataset combination and model based on BLEU score. On an author-constructed 'newtest' set, iterative BT raises BLEU from 29.67 to 40.25 (Eng2Lug) and from 32.92 to 44.25 (Lug2Eng), with TER decreasing. The abstract and Section 5 claim that this exceeds previous benchmarks by more than 10 BLEU units.

Significance. If the internal comparison is taken at face value, the paper provides a useful demonstration that iterative and incremental back translation can improve a very low-resource language pair, and it has several commendable practices: reporting SacreBLEU, ChrF2, and TER; documenting data sources in detail; and providing human-readable sample translations. The strongest contribution would be the curated and web-crawled datasets, although they are not currently released. The headline comparison with previous benchmarks is not established, and the selection protocol raises evaluation-bias concerns, so the paper's significance as a benchmark claim is currently limited.

major comments (4)
  1. [Abstract and Section 5] The claim that the authors exceed previous benchmarks by more than 10 BLEU units compares newtest scores (40.25 Eng2Lug, 44.25 Lug2Eng, Tables 6 and 7) with scores of 26.7 and 33.2 from [3], but there is no evidence that these numbers were computed on the same test set, with the same tokenization, or with the same BLEU implementation. The newtest set is new and unreleased, so the comparison is uncontrolled; either replace it with a comparison on a shared, released test set or remove the benchmark claim from the abstract.
  2. [Algorithm 1 and Section 4.3] The selection step in Algorithm 1 (lines 10–11) chooses monolingual dataset combinations and models 'based on the BLEU score,' and the paper then reports final BLEU scores on the same newtest set used to demonstrate the gains. If the same split was used for selection and final evaluation, the reported improvements are optimistically biased. The authors should state explicitly which split SelectBasedOnBLEU uses; ideally, selection should be performed on a held-out validation set, with the newtest set used only once for final reporting.
  3. [Section 4.3 and Section 4.1.1] The newtest set is described as 'focused on news from the local context' and is used to evaluate models whose monolingual training data include web-crawled material from the same news sources (Tables 2 and 3). Without a deduplication or overlap analysis, the gains may partly reflect domain match rather than the back-translation method itself. Please report the overlap between newtest and all training and monolingual sources and, if possible, evaluate on an independent out-of-domain test set.
  4. [Section 4.1 and Tables 6–7] The word 'significant' in the abstract is not supported by significance testing or variance estimates; all results appear to come from a single training run per condition. At minimum, the authors should report multiple seeds with standard deviations, or temper the wording of the claim.
minor comments (5)
  1. [Throughout] There are several typographical errors: 'la nguages' in the title, 'BLUE' for BLEU in Section 4.3, 'SacreBLE U' for SacreBLEU in Section 3.3, and 'reproducability' in Section 1.
  2. [Section 4.2] The large jump from 39.05 to 53.77 BLEU after removing the Bible text deserves a sentence of explanation; as written, it is surprising and could indicate an unexpected interaction between the default T/V split and the Bible-domain training data.
  3. [Section 1 and Section 4.3] The paper states that releasing datasets would aid reproducibility, but no URL or repository is provided for newtest or the crawled corpora; please include them or state clearly that they are not yet available.
  4. [References] References [29] and [30] appear to describe the same arXiv paper; they should be consolidated to avoid duplication.
  5. [Additional material] The arXiv source includes a 'sample-franklin.png' figure that is not referenced in the text; it should be removed or cited.

Circularity Check

1 steps flagged · score 6.0 of 10

OurBT selects monolingual datasets and models by BLEU on the same 'newtest' split that is later reported, so the reported gains are partly the selection objective rather than an independent prediction.

  1. fitted input called prediction [Section 3.2.2 (Algorithm 1, lines 10-11); Section 4.3 (Tables 6-7)]
    "Based on the BLEU score, OurBT identifies the most relevant datasets ... and models ... // Using the newtest dataset and applying our approach (OurBT) improved the model performance."

    Algorithm 1 chooses the best monolingual corpus combination and the best intermediate models 'Based on the BLEU score.' Section 4.3 then reports the BLEU scores of exactly those selected configurations on the newtest split, presenting the gains (6.27 Eng2Lug, 7.05 Lug2Eng) as evidence that OurBT improves translation. If the BLEU used for selection is computed on the same newtest set that is later reported—and the paper does not state that a separate validation split was used—the reported improvement is the maximization objective, not an independent estimate.

full rationale

The concrete circular step is the model/dataset selection on the evaluation metric: the same BLEU objective that picks the best combination is later reported as the improvement. Apart from this, the paper is largely self-contained: back translation is an established external technique, the Transformer and BPE are standard components, and no load-bearing uniqueness theorem or self-citation chain is invoked. The headline comparison to prior work (+10 BLEU over [3]) is a separate validity problem—the newtest set is not shared with [3], so the scores are not directly comparable—but that is a comparability issue, not a circular derivation. Because the central 'OurBT improves BLEU' result is partly the same BLEU objective used in selection, the circularity score is 6 rather than 0-2.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central result rests on standard NMT training assumptions and on the validity of comparing BLEU across different test sets. The main empirical choices are the best dataset combinations chosen by BLEU, which are fitted to the evaluation metric rather than fixed a priori.

free parameters (3)
  • Best Luganda monolingual dataset combination for Eng2Lug = Mozilla Common Voice, Makerere Text and Speech, YouTube news headlines
    Selected by BLEU in OurBT (Algorithm 1, Section 4.3); the reported Eng2Lug result depends on this score-driven choice.
  • Best English monolingual dataset combination for Lug2Eng = Digital Umuganda, Gamayun MiniKit, Chimpreports news
    Selected by BLEU in OurBT (Algorithm 1, Section 4.3); the reported Lug2Eng result depends on this score-driven choice.
  • Number of iterative BT steps = 3 to 4 iterations until convergence
    Convergence is judged by BLEU in Section 4.3; a different stopping rule could change the final scores.
assumptions (3)
  • domain assumption Back translation of monolingual data improves NMT quality for low-resource pairs
    Central mechanism taken from [28] and prior work; not re-derived in this paper.
  • domain assumption BLEU, ChrF2, and TER measured on the newtest split are valid and comparable measures of translation quality
    Used both for dataset selection in Algorithm 1 and for comparison with prior benchmarks in Section 5.
  • domain assumption The newtest set is representative of the evaluation conditions under which prior benchmarks were reported
    Necessary for the claim of beating previous work by more than 10 BLEU; the paper does not show test-set overlap with [3] or [8].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Augmentation With Back translation for Low Resource languages: A case of English and Luganda." pith.science (2026). https://pith.science/paper/53W7IJKY

@misc{pith2026250502463,
  author       = {Pith},
  title        = {Pith review of: Data Augmentation With Back translation for Low Resource languages: A case of English and Luganda},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53W7IJKY}},
  note         = {Machine review of arXiv:2505.02463}
}
read the original abstract

In this paper,we explore the application of Back translation (BT) as a semi-supervised technique to enhance Neural Machine Translation(NMT) models for the English-Luganda language pair, specifically addressing the challenges faced by low-resource languages. The purpose of our study is to demonstrate how BT can mitigate the scarcity of bilingual data by generating synthetic data from monolingual corpora. Our methodology involves developing custom NMT models using both publicly available and web-crawled data, and applying Iterative and Incremental Back translation techniques. We strategically select datasets for incremental back translation across multiple small datasets, which is a novel element of our approach. The results of our study show significant improvements, with translation performance for the English-Luganda pair exceeding previous benchmarks by more than 10 BLEU score units across all translation directions. Additionally, our evaluation incorporates comprehensive assessment metrics such as SacreBLEU, ChrF2, and TER, providing a nuanced understanding of translation quality. The conclusion drawn from our research confirms the efficacy of BT when strategically curated datasets are utilized, establishing new performance benchmarks and demonstrating the potential of BT in enhancing NMT models for low-resource languages.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Saturation Point of Backtranslation in High Quality Low Resource English Gujarati Machine Translation

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Backtranslated Gujarati-English data fails to improve a strong MBART50 baseline trained on 50k parallel pairs, slightly reducing BLEU from 43.8 to 43.0.

Reference graph

Works this paper leans on

36 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [3]

    Benjamin Akera, Jonathan Mukiibi, Lydia Sanyu Naggayi, Claire Babirye, Isaac Owomugisha, Solomon Nsumba, Joyce Na katumba-Nabende, En- gineer Bainomugisha, Ernest Mwebaze, and John Quinn. 2022. Machine translation for african languages: Community crea tion of datasets and models in uganda. In 3rd Workshop on African Natural Language Processing

  2. [1]

    David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azim e, Jesujoba Oluwadara Alabi, Atnafu Lambebo Tonja, Christi ne Mwase, Odunayo Ogun- depo, Bonaventure FP Dossou, Akintunde Oladipo, Doreen Nix dorf, et al. 2023. Masakhanews: News topic classification fo r african languages. arXiv preprint arXiv:2304.09972 (2023)

  3. [2]

    Alham Fikri Aji and Kenneth Heafield. 2020. Fully synthet ic data improves neural machine translation with knowledge distillation. arXiv preprint arXiv:2012.15455 (2020)

  4. [4]

    Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi D ou, Marcello Federico, Christian Federman, Dmitriy Genzel , Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, et al. 2020. TICO-19: the tran slation initiative for COvid-19. arXiv preprint arXiv:2007.01788 (2020)

  5. [5]

    Claire Babirye, Joyce Nakatumba-Nabende, Andrew Katum ba, Ronald Ogwang, Jeremy Tusubira Francis, Jonathan Mukii bi, Medadi Ssentanda, Lilian D Wanzare, and Davis David. 2022. Building text and sp eech datasets for low resourced languages: A case of languag es in east Africa. (2022)

  6. [6]

    Claire Babirye, Jeremy Tusubira, Jonathan Mukiibi, Joy ce Nakatumba-Nabende, and Andrew Katumba. 2023. Sentiment Tagged Parallel Corpus for Luganda and Swahili. https://doi.org/10.7910/DVN/XSGIKR

  7. [7]

    Elizabeth Baertlein and Martin Ssekitto. 2014. Luganda Nouns: Inflectional Morphology and Tests. Linguistic Portfolios 3, 1 (2014), 3. 10 Data Augmentation With Back translation for Low Resource lan guages: A case of English and Luganda NLPIR 2024, December 13–15,2024, Okayama, Japan

  8. [8]

    Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat , Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, et al. 2022. Building machine translation system s for the next thousand languages. arXiv preprint arXiv:2205.03983 (2022)

Show all 36 references
  1. [9]

    Shweta Chauhan, Shefali Saxena, and Philemon Daniel. 20 22. Improved unsupervised neural machine translation with semantically weighted back translation for morphologically rich and low resource lang uages. Neural Processing Letters 54, 3 (2022), 1707–1726

  2. [10]

    Yong Cheng and Yong Cheng. 2019. Joint training for pivo t-based neural machine translation. Joint training for neural machine translation (2019), 41–54

  3. [11]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vis hrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cro ss-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)

  4. [12]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangi er. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381 (2018)

  5. [13]

    Turkic Morph eme

    Ayrat Gatiatullin, Lenara Kubedinova, Nikolai Prokop yev, and Abduramanov Ibraim. 2022. Toolset of “Turkic Morph eme” Portal for Creation of Electronic Corpora of Turkic Languages in a Unified Conceptu al Space. In 2022 7th International Conference on Computer Science and E ngi...

  6. [14]

    Catherine Gitau and Vukosi Marivate. 2023. Textual Aug mentation Techniques Applied to Low Resource Machine Trans lation: Case of Swahili. arXiv preprint arXiv:2306.07414 (2023)

  7. [15]

    Unmesh Gundecha. 2015. Selenium Testing Tools Cookbook. Packt Publishing Ltd

  8. [16]

    Krupakar Hans and RS Milton. 2016. Improving the perfor mance of neural machine translation involving morphologic ally rich languages. arXiv preprint arXiv:1612.02482 (2016)

  9. [17]

    Cong Duy Vu Hoang, Philipp Koehn, Gholamreza Haffari, an d Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In 2nd Workshop on Neural Machine Translation and Generation . Association for Computational Linguistics, 18–24

  10. [18]

    Imran Khan Mohd Jais, Amelia Ritahani Ismail, and Syed Q amrun Nisa. 2019. Adam optimization algorithm for wide and d eep neural network. Knowl. Eng. Data Sci. 2, 1 (2019), 41–46

  11. [19]

    Chang Jin, Shigui Qiu, Nini Xiao, and Hao Jia. 2022. AdMi x: A mixed sample data augmentation method for neural machin e translation. arXiv preprint arXiv:2205.04686 (2022)

  12. [20]

    Vanlalmuansangi Khenglawt, Sahinur Rahman Laskar, Pa rtha Pakray, and Ajoy Kumar Khan. [n. d.]. Addressing data sc arcity issue for English– Mizo neural machine translation using data augmentation an d language model. Journal of Intelligent & Fuzzy Systems Preprint ([n. d.]), 1–11

  13. [21]

    Richard Kimera, Daniela N Rim, and Heeyoul Choi. 2022. B uilding a Parallel Corpus and Training Translation Models B etween Luganda and English. Information Science Journal 49, 11 (2022), 1009–1016

  14. [22]

    Annie Lamar and Zeyneb Kaya. 2023. Measuring the Impact of Data Augmentation Methods for Extremely Low-Resource NM T. In Proceedings of the The Sixth Workshop on Technologies for Machine Translat ion of Low-Resource Languages (LoResMT 2023) . 101–109

  15. [23]

    Maxwell Mojapelo and Jan Buys. 2023. Data augmentation for low resource neural machine translation for sotho-tswa na languages. (2023)

  16. [24]

    Jonathan Mukiibi, Claire Babirye, Jeremy Tusubira, To bias Bateesa, Eric Peter Wairagala, Chodrine Mutebi, Joyce Nakatumba-Nabende, Andrew Katumba, Ivan Ssenkungu, and Medadi Sentanda. 2023. Lugand a Monolingual Corpus. https://doi.org/10.7910/DVN/EQOWTW

  17. [25]

    Jonathan Mukiibi, Andrew Katumba, Joyce Nakatumba-Na bende, Ali Hussein, and Josh Meyer. 2022. The makerere radiospeech corpus: A Luganda radio corpus for automatic speech recognition. arXiv preprint arXiv:2206.09790 (2022)

  18. [26]

    Seokjin Oh, Woohwan Jung, et al. 2023. Data augmentatio n for neural machine translation using generative language model. arXiv preprint arXiv:2307.16833 (2023)

  19. [27]

    Daniela N Rim, Richard Kimera, and Heeyoul Choi. 2023. M ini-Batching with Similar-Length Sentences to Quickly Train NMT Models. Information Science Journal 50, 7 (2023), 614–620

  20. [28]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015 . Improving neural machine translation models with monolin gual data. arXiv preprint arXiv:1511.06709 (2015)

  21. [30]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 . Neural Machine Translation of Rare Words with Subword Units. arXiv: 1508.07909 [cs.CL]

  22. [31]

    AM Ssentumbwe, BM Kim, and HA Lee. 2019. English to Lugan da SMT: Ganda Noun Class Prefix Segmentation for Enriched Mac hine Translation [J]. International Journal of Advanced Trends in Computer Scien ce and Engineering 8, 5 (2019), 1861–1868

  23. [32]

    Amane Sugiyama and Naoki Yoshinaga. 2019. Data augment ation using back-translation for context-aware neural machine translation. In Proceed- ings of the fourth workshop on discourse in machine translat ion (DiscoMT 2019) . 35–44

  24. [33]

    Herry Sujaini, Samuel Cahyawijaya, and Arif B Putra. 20 23. Analysis of Language Model Role in Improving Machine Tra nslation Accuracy for Extremely Low Resource Languages. Journal of Advances in Information Technology 14, 5 (2023)

  25. [34]

    Mengtao Sun, Hao Wang, Mark Pasquine, and Ibrahim A. Ham eed. 2021. Machine translation in low-resource languages b y an adversarial neural network. Applied Sciences 11, 22 (2021), 10860

  26. [35]

    Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT –building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Transla tion. 479–480

  27. [36]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Po losukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). 11 NLPIR 2024, December 13–15, 2024, Okayama, Japan Rich...

  28. [37]

    acm-jdslogo.png

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau , Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and E douard Grave. 2019. CCNet: Extracting high quality monolingual datasets from w eb crawl data. arXiv preprint arXiv:1911.00359 (2019). 12 This figure "acm-jdslogo.pn...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.