{"id":"f51267b0-f8ae-4cc0-afe8-ff019d4d7566","arxiv_id":"2505.02463","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Back translation with dataset selection raises English-Luganda NMT BLEU scores by about 10 points on a new test set, though the comparison with previous benchmarks is not on a shared test set.","lead":"This paper applies back translation, a data augmentation technique, to English and Luganda machine translation, and reports BLEU-score gains over prior work on a new test set. It also proposes a custom approach that selects the best monolingual datasets for training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '+10 BLEU over prior benchmarks' claim rests on comparing scores on a new, unreleased test set to scores from prior work on different test sets; without a shared test set or released data the claim is not established.","rationale":"The reader's weakest assumption correctly identifies the decisive flaw: the '+10 BLEU' headline claim compares scores on a self-constructed, unreleased newtest with scores from prior work on unknown test sets. BLEU is not comparable across test sets, and the paper provides no calibration, matched test split, or released data to make the comparison valid. This concern is load-bearing because the abstract and conclusion rest entirely on this cross-paper comparison; without it, the contribution reduces to an internally consistent but uncontrolled demonstration that back translation helps on one particular test set. I agree with the reader's CONDITIONAL verdict because the internal experiments are plausible and the dataset construction is potentially useful, but the central benchmark claim is not currently verifiable. My addition of the selection-on-newtest ambiguity strengthens rather than replaces the reader's concern: if Algorithm 1's BLEU-based selection used the same newtest split as final evaluation, even the internal improvements are optimistically biased. A concrete check on a shared or public test set would settle the issue, so the appropriate verdict remains CONDITIONAL rather than REJECT or ACCEPT.","tokens_in":12285,"tokens_out":4786,"duration_ms":59102,"concrete_test":"Obtain the test splits used by Akera et al. [3] (or, if unavailable, use a public Luganda-English benchmark such as FLORES-200) and run the final Eng2Lug and Lug2Eng models from this paper on that exact data with a single SacreBLEU configuration. If the gap over the corresponding prior systems remains greater than 10 BLEU in both directions, the benchmark claim survives; if it shrinks or reverses, the reported gain is a test-set artifact. Additionally, report which split Algorithm 1 used for dataset/model selection; if it was newtest, re-run selection on validation only and re-report final scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('exceeding previous benchmarks by more than 10 BLEU score units across all translation directions') is supported only by comparing the final newtest scores (40.25 Eng2Lug and 44.25 Lug2Eng, Tables 6-7) with previously reported scores from [3] (26.7 and 33.2) quoted in Section 5. No evidence is given that these numbers were produced on the same test set, the same tokenization/BLEU implementation, or comparable data splits. Section 4.3 introduces newtest as a set constructed by the authors to be 'more general' and 'focused on news'; it is not released and is not shown to be comparable in difficulty to the test sets used in [3]. The paper itself states in Section 1 that prior monolingual datasets are not publicly available and that releasing data would aid reproducibility 'a component that is not possible today,' directly acknowledging that the benchmark comparison cannot currently be checked. Additionally, the OurBT procedure (Algorithm 1, lines 10-11) selects monolingual datasets and models 'based on the BLEU score,' and Section 4.3 reports those selections on newtest; if the same newtest split was used for selection and final evaluation, the reported gains are optimistically biased. Either way, the headline improvement over prior work is not supported by a controlled comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies back translation (BT) for English–Luganda neural machine translation. It assembles bilingual data from four sources and monolingual data for both languages, including web-crawled news text, trains Transformer models, and applies standard, incremental, iterative, and a proposed 'OurBT' variant. OurBT (Algorithm 1) selects the best monolingual dataset combination and model based on BLEU score. On an author-constructed 'newtest' set, iterative BT raises BLEU from 29.67 to 40.25 (Eng2Lug) and from 32.92 to 44.25 (Lug2Eng), with TER decreasing. The abstract and Section 5 claim that this exceeds previous benchmarks by more than 10 BLEU units.","tokens_in":12581,"tokens_out":6084,"duration_ms":74635,"significance":"If the internal comparison is taken at face value, the paper provides a useful demonstration that iterative and incremental back translation can improve a very low-resource language pair, and it has several commendable practices: reporting SacreBLEU, ChrF2, and TER; documenting data sources in detail; and providing human-readable sample translations. The strongest contribution would be the curated and web-crawled datasets, although they are not currently released. The headline comparison with previous benchmarks is not established, and the selection protocol raises evaluation-bias concerns, so the paper's significance as a benchmark claim is currently limited.","major_comments":[{"comment":"The claim that the authors exceed previous benchmarks by more than 10 BLEU units compares newtest scores (40.25 Eng2Lug, 44.25 Lug2Eng, Tables 6 and 7) with scores of 26.7 and 33.2 from [3], but there is no evidence that these numbers were computed on the same test set, with the same tokenization, or with the same BLEU implementation. The newtest set is new and unreleased, so the comparison is uncontrolled; either replace it with a comparison on a shared, released test set or remove the benchmark claim from the abstract.","section":"Abstract and Section 5"},{"comment":"The selection step in Algorithm 1 (lines 10–11) chooses monolingual dataset combinations and models 'based on the BLEU score,' and the paper then reports final BLEU scores on the same newtest set used to demonstrate the gains. If the same split was used for selection and final evaluation, the reported improvements are optimistically biased. The authors should state explicitly which split SelectBasedOnBLEU uses; ideally, selection should be performed on a held-out validation set, with the newtest set used only once for final reporting.","section":"Algorithm 1 and Section 4.3"},{"comment":"The newtest set is described as 'focused on news from the local context' and is used to evaluate models whose monolingual training data include web-crawled material from the same news sources (Tables 2 and 3). Without a deduplication or overlap analysis, the gains may partly reflect domain match rather than the back-translation method itself. Please report the overlap between newtest and all training and monolingual sources and, if possible, evaluate on an independent out-of-domain test set.","section":"Section 4.3 and Section 4.1.1"},{"comment":"The word 'significant' in the abstract is not supported by significance testing or variance estimates; all results appear to come from a single training run per condition. At minimum, the authors should report multiple seeds with standard deviations, or temper the wording of the claim.","section":"Section 4.1 and Tables 6–7"}],"minor_comments":[{"comment":"There are several typographical errors: 'la nguages' in the title, 'BLUE' for BLEU in Section 4.3, 'SacreBLE U' for SacreBLEU in Section 3.3, and 'reproducability' in Section 1.","section":"Throughout"},{"comment":"The large jump from 39.05 to 53.77 BLEU after removing the Bible text deserves a sentence of explanation; as written, it is surprising and could indicate an unexpected interaction between the default T/V split and the Bible-domain training data.","section":"Section 4.2"},{"comment":"The paper states that releasing datasets would aid reproducibility, but no URL or repository is provided for newtest or the crawled corpora; please include them or state clearly that they are not yet available.","section":"Section 1 and Section 4.3"},{"comment":"References [29] and [30] appear to describe the same arXiv paper; they should be consolidated to avoid duplication.","section":"References"},{"comment":"The arXiv source includes a 'sample-franklin.png' figure that is not referenced in the text; it should be removed or cited.","section":"Additional material"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a workshop-style case study and is plausibly within scope for an NLP venue, but its framing as establishing new performance benchmarks is not supported by the experimental protocol. The absence of a released test set and the uncontrolled comparison are disqualifying for that framing; after revision, the paper would be more appropriate as a case study of iterative back translation for Luganda than as a benchmark paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good to get your read on this one. I mostly agree with your take, with a couple of qualifications.\n\nWhat's actually new: they've assembled a decent-size English–Luganda parallel corpus from public sources (roughly 98k segments including Bible, TICO-19, etc.) and crawled additional news-domain monolingual text, and they show that iterative back translation gives consistent BLEU gains on their own \"newtest\" split (29.67→40.25 Eng→Lug; 32.92→44.25 Lug→Eng), with TER dropping. That internal result is coherent, and the use of SacreBLEU, ChrF2, and TER is the right instinct for a morphologically rich language. The dataset contribution, if actually released, would be the real value here.\n\nThe soft spots are where you put them. The headline claim—\"exceeding previous benchmarks by more than 10 BLEU\"—compares numbers on a newly built, unreleased news-focused test set against numbers reported by Akera et al. on whatever test set they used. That is apples-to-oranges unless the sets are shown to be comparable, and the paper doesn't do that. The paper even says in the introduction that prior monolingual datasets aren't public and releasing data would help reproducibility \"a component that is not possible today,\" which is a tacit admission that the benchmark comparison can't currently be checked.\n\nThere's also a selection-on-the-test-set worry. Algorithm 1 picks the best monolingual dataset combination and iteration count \"based on the BLEU score,\" and the paper then reports those same BLEU numbers. If the selection was done on newtest, the reported gains are optimistically biased. The paper never says a separate validation set was used for selection. That's a real methodological gap, though it's fixable with a clearer protocol or a proper held-out set.\n\nMinor but still worth flagging: the \"BLEU\" column and \"SacreBLEU\" column clearly differ (e.g., 29.67 vs 28.3), so the implementation behind the headline BLEU numbers is ambiguous. And no code or data is released despite the stated goal of alleviating data scarcity.\n\nNet: the internal finding—iterative BT helps on a harder news test set—is plausible and worth having. The comparative claim against prior work is not established, and the selection protocol needs to be stated explicitly. This deserves referee time because the low-resource language pair and dataset effort matter, but it should come back with major revisions: release the data, compare against prior work on a shared or released test set, and report the selection procedure with a validation set that is not the test set.\n\nRecommendation: send to peer review, with the expectation of heavy revision.","headline":"Useful data effort and a plausible internal result, but the +10 BLEU claim over prior work is not established and the selection step risks overfitting to the test set.","tokens_in":13127,"tokens_out":4308,"would_cite":false,"duration_ms":50163,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selective back translation lifts English–Luganda BLEU by more than 10 points, the paper reports.","keywords":["Neural Machine Translation","Back Translation","Low Resource Languages","English-Luganda Language Pair","Data Augmentation","Translation Quality Metrics","Iterative Back Translation","Incremental Back Translation"],"falsifier":"Run the earlier back-translation system [3] on this paper's 'newtest' sentences and compute its BLEU; if it scores within 10 BLEU points of 40.25 (Eng2Lug) or 44.25 (Lug2Eng), the central comparative claim is not supported. The mirror-image check is to run this paper's final model on the earlier work's test set and see whether the gap survives.","tokens_in":1699,"feed_emoji":"🗣️","tokens_out":1737,"duration_ms":113859,"temperature":0.7,"pith_summary":"This paper tries to show that back translation—translating monolingual text with a preliminary model to create synthetic parallel sentences—can give a large boost to neural machine translation for a low-resource language pair, English and Luganda. The authors build bilingual and monolingual datasets from public and web-crawled sources, then run iterative and incremental back translation with a dataset-selection step they call OurBT. They report final BLEU scores of 40.25 for English-to-Luganda and 44.25 for Luganda-to-English on their newly built 'newtest' set, more than 10 BLEU points above earlier back-translation results for this pair. If the comparison holds, it would mean a practical recipe for improving low-resource translation using small, fragmented monolingual corpora, and the paper releases the datasets to make the result reproducible. The claim depends on the new test set being comparable to earlier evaluations.","feed_headline":"Back translation lifts English-Luganda BLEU by 10+ points","feed_subtitle":"Picking small context-matched monolingual datasets beats one-shot back translation for a low-resource pair.","key_machinery":"The central mechanism is the paper's 'OurBT' algorithm: starting from a bilingual baseline model, it translates each small monolingual dataset in both directions, retrains a model on each resulting synthetic set, ranks the datasets by the BLEU score of the retrained models, and then retrains the final model on synthetic translations of the top-ranked monolingual sets. Iterating this cycle implements iterative back translation; adding monolingual data in growing portions implements incremental back translation. The mechanism acts as a selection filter: it scores fragmented monolingual corpora by how much they help the translation task, and the paper finds that the best-scoring sets are those with contextual similarity to the training data.","core_discovery":"The paper's central discovery claim is that a back-translation routine that selects the best-performing small monolingual datasets produces large gains for English–Luganda: BLEU rises from 29.67 to 40.25 for English-to-Luganda and from 32.92 to 44.25 for Luganda-to-English on the newtest split, more than 10 points above the earlier back-translation scores of 26.7 and 33.2 [3]. The paper also claims that the improvement is not an artifact of BLEU alone, since ChrF2 rises and TER falls in both directions, and sample translations align with reference translations better than a commercial online translator. It attributes the gains to selecting datasets whose context matches the training data rather than to simply adding more monolingual text.","pith_inferences":["The 10-point margin over prior work is only meaningful if the test sets are comparable; a head-to-head evaluation on a shared held-out set could confirm or shrink the gap.","The fact that context-matching datasets win selection suggests back translation here is doing implicit domain adaptation, so a cheaper selector could rank monolingual datasets by domain similarity without retraining a full model per candidate.","The recipe—small monolingual corpora from varied sources, scored one at a time—should transfer to other Bantu and low-resource languages with fragmented data; testing it on Swahili, Acholi, or Runyankore would show whether the selection step generalises.","The large gap between the default validation scores and the newtest scores (for example, 53.77 versus 29.67 for Eng2Lug) shows how much test-set choice affects apparent quality; future benchmarks for this language pair should settle on a shared public test split to make comparisons meaningful."],"forward_implications":["English–Luganda translation can reach BLEU scores of 40.25 and 44.25 on the newtest split using back translation over small, selected monolingual datasets, a level that would make the pair substantially more usable.","Dataset selection matters: applying standard back translation to the best dataset combination gives only +2.49 BLEU for Eng2Lug and +0.45 for Lug2Eng, while the selection-then-retrain routine gives +6.27 and +7.05 before iteration.","Repeating the selection-retrain cycle improves scores in each of three reported iterations, suggesting the routine can be run until convergence rather than fixed at a single pass.","The publicly released bilingual and monolingual datasets give other researchers a common resource for English–Luganda NMT, addressing the previous lack of public data for this pair.","Evaluation with ChrF2 and TER gives a fuller picture for a morphologically rich language: TER falls from 69.7 to 65.4 for Eng2Lug and from 64.7 to 60.4 for Lug2Eng, consistent with the BLEU gains."],"supporting_citations":[{"why":"Provides the previous English–Luganda back-translation benchmark (26.7 and 33.2 BLEU) that the paper claims to surpass by more than 10 points.","marker":"[3]"},{"why":"Supplies the main publicly available bilingual dataset (41,070 parallel sentences) used as training data.","marker":"[21]"},{"why":"Introduces back translation, the core data-augmentation technique the paper extends.","marker":"[28]"},{"why":"Defines iterative and incremental back translation, the two variants the paper applies and combines.","marker":"[17]"},{"why":"Provides prior large-scale Luganda back-translation work and the ChrF evaluation argument for morphologically rich languages.","marker":"[8]"},{"why":"Provides the Transformer base architecture used for all NMT models in the experiments.","marker":"[36]"},{"why":"Used to justify beam-search generation of synthetic data during back translation.","marker":"[12]"}],"fun_headline_variants":["Smarter back translation lifts English-Luganda by 10+ BLEU","Context-matched data drives 10+ BLEU gain for low-resource NMT","Iterative back translation sets new English-Luganda record","English-Luganda MT leaps 10+ BLEU with selective back translation"],"cache_read_input_tokens":15232,"weakest_assumption_plain":"The headline claim assumes that BLEU scores on the newly built 'newtest' set can be compared directly with earlier reported BLEU scores obtained on different test sets; if the test sets differ, the claimed 10-point lead is not established.","fun_headline_variants_meta":{"raw":{"variants":["Smarter back translation lifts English-Luganda by 10+ BLEU","Context-matched data drives 10+ BLEU gain for low-resource NMT","Iterative back translation sets new English-Luganda record","English-Luganda MT leaps 10+ BLEU with selective back translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2416,"prompt_tokens":920,"completion_tokens":1496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1413}},"tokens_in":536,"tokens_out":1496,"duration_ms":12867,"temperature":1.0,"reasoning_tokens":1413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:50:00.150033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the earlier back-translation system [3] on this paper's 'newtest' sentences and compute its BLEU; if it scores within 10 BLEU points of 40.25 (Eng2Lug) or 44.25 (Lug2Eng), the central comparative claim is not supported. The mirror-image check is to run this paper's final model on the earlier work's test set and see whether the gap survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the previous English–Luganda back-translation benchmark (26.7 and 33.2 BLEU) that the paper claims to surpass by more than 10 points."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the main publicly available bilingual dataset (41,070 parallel sentences) used as training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines iterative and incremental back translation, the two variants the paper applies and combines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Transformer base architecture used for all NMT models in the experiments."}],"review_version":1}