{"id":"934ce1e2-a250-4d3f-8e34-f99291407452","arxiv_id":"2505.15069","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Bandit-based model selection matches or slightly improves on the best single NMT system for in-domain English-to-African translation, but the claimed high-confidence statistical support is absent.","lead":"Sony Research India tested four multi-armed bandit algorithms (UCB, LinUCB, Thompson Sampling, Neural LinUCB) that choose between five pre-trained translation models sentence by sentence for English-to-African language translation. They report the bandit selection can match or slightly beat the best single model on in-domain BLEU, even when no reference translation is available.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.68% UCB improvement is not established: the paper never shows the test sentences were disjoint from bandit exploration and hyperparameter tuning, gives no significance tests, and Table 2's own averages contradict the 'UCB performs best' claim.","rationale":"The paper's core idea, applying multi-armed bandits to low-resource model selection, is plausible and the algorithms are standard. The problem is not the method class but the evidence for the headline number. The reader's weakest assumption, that the evaluation uses a genuinely held-out test set, is exactly the load-bearing concern I would raise. The hyperparameter section strongly suggests that the 'rest of the remaining sentences' from the validation set were used for testing, with no explicit statement that the 1,000 test samples in §5 are separate from the bandit's reward/exploration data. This potential contamination directly inflates any comparison to a static NLLB baseline. The absence of confidence intervals or significance tests compounds the issue: the 2.68% relative gain is dominated by a single cell, and a paired bootstrap might well include zero. The Table 2 target-free result is an additional internal inconsistency that reinforces the need for a clean re-evaluation. Because the central claim is a small quantitative advantage and the current evidence does not establish it, the reader's REJECT verdict remains appropriate; a revised paper with an explicit disjoint split, frozen policy at test time, and bootstrap intervals could move to CONDITIONAL or ACCEPT.","tokens_in":10193,"tokens_out":7270,"duration_ms":62533,"concrete_test":"Ask the authors for the exact split indices and evaluation script. Then re-run every algorithm under a strict protocol: all hyperparameters and bandit convergence on a validation split only; freeze the policy before scoring; evaluate on a disjoint 1,000-sentence test split never used for reward computation. Report paired bootstrap 95% confidence intervals for UCB minus NLLB BLEU per cell and for the average. If the 2.68% relative gain falls inside the interval, or if the Table 2 ranking changes as in the simple average above, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (UCB outperforms NLLB by 2.68% relative BLEU, §5) depends on the reported test BLEU being computed on a held-out set that the bandit never saw. The paper does not establish this. §4 says 'We sample 1,000 parallel samples for validation (seed data for model convergence) and testing each,' but the hyperparameter section says 'initial few sentences of validations were used for convergence of algorithm and the algorithm was tested on rest of the remaining sentences.' Read literally, one validation set is split into a small convergence prefix and a 'rest' that serves as the test set; there is no statement that the 1,000 test sentences used in §5 are disjoint from the sentences on which bandit rewards (BLEU+COMET) were observed. If the bandit updated on the same sentences later scored, the selection is fitted to the test set and the comparison to static NLLB is biased. The 2.68% average is also fragile: it is almost entirely driven by Religious-Swahili (+4.35), while News-Yoruba is negative (−0.13), and no confidence interval or significance test is reported. In the target-free scenario, the paper claims 'UCB performs the best,' but averaging the nine BLEU entries in Table 2 gives TS 20.84, UCB 20.72, LinUCB 20.65, NL 19.99, contradicting the prose and weakening the claimed robustness. These are internal consistency and validation-hygiene problems, not disagreements with community consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using multi-armed bandit algorithms (UCB, LinUCB, Neural LinUCB, and Thompson Sampling) to select among five pretrained machine-translation systems for English-to-Yoruba, English-to-Swahili, and English-to-Igbo translation in the News, Movies, and Religious domains. Rewards are defined as a weighted combination of BLEU and reference-based COMET when parallel data is available, and as CometKiwi in a target-free scenario. The authors claim that UCB outperforms the best static system, NLLB, by an average of 2.68% BLEU in the parallel-data case and is also the best algorithm in the target-free scenario.","tokens_in":10511,"tokens_out":8088,"duration_ms":62926,"significance":"The problem addressed is relevant: selecting an adequate translation model for low-resource languages and narrow domains without fine-tuning is practically important, and treating model selection as a contextual bandit is a plausible, lightweight strategy. The paper evaluates on three African languages and three domains, includes both target-available and target-free reward settings, and compares several standard bandit algorithms. If the results were established on a properly held-out test set with significance testing, the contribution would be useful for practitioners. As it stands, however, the manuscript does not provide such evidence, and several internal inconsistencies prevent acceptance in its current form.","major_comments":[{"comment":"The evaluation protocol does not establish that the 1,000 sentences scored in Tables 1 and 2 are disjoint from the sentences on which the bandits observed rewards or on which hyperparameters were tuned. Section 4 states, \"initial few sentences of validations were used for convergence of algorithm and the algorithm was tested on rest of the remaining sentences,\" and §5 says performance is evaluated using 1,000 test samples without stating their provenance. If the same sentences were used for exploration rewards (BLEU+COMET in the parallel-data case) and for final BLEU scoring, the comparison against static NLLB is in-sample and the reported 2.68% gain is not a valid estimate. The authors must specify the exact train/validation/test split, state explicitly that no reward was observed on the final test sentences, and report results on a genuinely held-out set.","section":"§4 and §5"},{"comment":"The prose claim that UCB performs best in the target-free scenario is contradicted by the table's own averages. Averaging the nine BLEU entries in Table 2 gives Thompson Sampling 20.84, UCB 20.72, LinUCB 20.65, and Neural LinUCB 19.99. In addition, in Table 1, UCB ties NLLB in five of nine cells and is slightly worse in News-Yoruba (9.539 vs. 9.67); the 2.68% average improvement is driven almost entirely by the Religious-Swahili cell (+4.35). The claim of \"superior performance across all languages and domains\" is therefore an overstatement. The prose should be reconciled with the tables or the claims should be qualified to report a single average gain with appropriate caveats.","section":"Table 2 and §5"},{"comment":"The abstract and Figure 1(c) promise \"optimal model selection with high confidence\" and statistically significant results, but no significance tests, confidence intervals, or run-to-run variance are reported anywhere in the manuscript. With average differences of a few tenths of a BLEU point over 1,000 sentences, the observed gaps (e.g., News-Igbo +0.10 in Table 1) are within typical noise. The authors should provide paired bootstrap confidence intervals or equivalent significance tests for the headline comparisons, and report the number of independent runs and the variance across runs.","section":"§1, Figure 1(c), and §5"},{"comment":"The reward design is partly circular with respect to the evaluation metric. When parallel data is present, the reward is λ·BLEU + (1−λ)·COMET, and the same BLEU metric is used for the final evaluation in Tables 1 and 2. Since λ=0.4 was selected by \"hit and trial\" on the validation set (§4), the comparison is in-sample with respect to the reward design. The authors should either use a reward that excludes the evaluation metric on the test sentences, or demonstrate that the conclusions are stable across a range of λ values and on a strictly held-out test set.","section":"§3 and §4"}],"minor_comments":[{"comment":"The sentence beginning \"As a result, selecting an optimal model...\" is grammatically incomplete and should be revised; \"As a result\" should be capitalized.","section":"Abstract"},{"comment":"The hyperparameter table is incomplete: UCB's λ is listed as a range \"[0.4-0.6]\" while the text says λ=0.4 gives the best results; the exact values and the tuning procedure should be reported for every algorithm.","section":"Table 3"},{"comment":"The tables use inconsistent decimal precision (e.g., 3.847, 7.087, 9.539); all BLEU scores should be reported with the same number of decimal places.","section":"Tables 1 and 2"},{"comment":"The figure caption labels panel (c) as \"RL algo.\" and the panels are not clearly referenced in the text; a fuller caption and explicit in-text references would improve readability.","section":"Figure 1"},{"comment":"The reference list contains duplicate entries (Saunders 2022a and 2022b) and malformed author names (e.g., \"Ricardo et al. Rei\", \"et al., 2024b\"); these should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the core idea, and the experimental design is reasonable in principle. The main problems are evaluation hygiene and internal consistency rather than a fundamentally flawed method. With a clean held-out split, significance testing, and a careful rewrite of the claims, the paper could become a useful contribution. Given the size of the required re-analysis, I recommend major revision rather than outright rejection, but the revision must be substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing to know: this is a small, clearly written empirical study with a plausible idea and an unsupported central claim. The authors apply four off-the-shelf bandit algorithms (UCB, LinUCB, Thompson Sampling, Neural LinUCB) to pick among five NMT models for English-to-Yoruba/Swahili/Igbo across News/Movies/Religious domains, and add a target-free setting that uses CometKiwi as reward. That combination is new as far as I know, and the paper is easy to follow.\n\nWhat it does well: the setup is honest in its aims, the pool of models is realistic, and the target-free reward is a genuinely useful trick for low-resource settings. If the numbers held up, small-sample bandit selection would be a practical contribution.\n\nThe soft spots are significant. First, the central claim—UCB beats NLLB by 2.68% relative BLEU—is fragile: it's driven almost entirely by one cell (Religious Swahili, +4.35). Excluding that cell, the average difference is under half a percent. Second, there are no significance tests or confidence intervals anywhere, despite the paper's own motivation being 'high confidence' and statistical reliability. Third, the evaluation split is ambiguous: §4 says 1,000 samples for validation and testing, and then says the algorithm was tested on the 'rest of the remaining sentences' after a few validation sentences for convergence. Read literally, the test set may overlap with sentences on which the bandit observed rewards. That would bias the comparison against static NLLB. The authors need to state clearly that the 1,000 test sentences were never seen by the bandit during exploration. Fourth, the target-free results contradict the prose: Table 2's averages give Thompson Sampling 20.84 vs UCB 20.72, so 'UCB performs the best' is not supported even by their own table.\n\nThe circularity concern about λ tuned on validation is minor but real; it's hyperparameter selection, not a fatal flaw.\n\nWho is this for? People working on low-resource MT and model selection might get a useful idea from the framework. But as it stands, the experimental grounding doesn't back the claims. This is a rejected paper in its current form, though the idea could be salvaged with a rigorous held-out evaluation, significance testing, and corrected prose. I don't think it warrants a serious referee yet; I'd send it back for major revision or rejection.","headline":"A plausible bandit-based model selection idea for low-resource MT, but the central 2.68% claim collapses under scrutiny and the evaluation hygiene is not there.","tokens_in":11105,"tokens_out":4914,"would_cite":false,"duration_ms":35901,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a UCB multi-armed bandit can select the best machine-translation model for in-domain African-language data, outperforming the strongest individual model by an average of 2.68% BLEU, and that the selection works even…","keywords":["machine translation","low-resource languages","African languages","domain adaptation","multi-armed bandits","contextual bandits","model selection","BLEU"],"falsifier":"Re-run the procedure with a strictly separated validation set and a fresh 1,000-sentence test set for each language-domain pair, and check whether UCB still beats the best single model by the reported margin with overlapping confidence intervals.","tokens_in":9972,"feed_emoji":"🎯","tokens_out":9589,"duration_ms":67027,"temperature":0.7,"pith_summary":"This paper tries to establish that a multi-armed bandit, specifically the Upper Confidence Bound algorithm, can pick the best machine-translation model for a given domain and language when in-domain data is scarce. It reports that this selection approach beats the strongest individual model, NLLB, by an average of 2.68% BLEU across three African languages and three domains, and that the same method works when no reference translations exist. The payoff would be a way to adapt to a domain without fine-tuning and without large parallel corpora, which matters for low-resource languages where such data rarely exists.","feed_headline":"Bandit algorithm beats best African translation model by 2.68% BLEU","feed_subtitle":"Multi-armed bandits choose the best NMT model with only 1,000 sentences, even without reference translations.","key_machinery":"The central machinery is the multi-armed bandit framing: the arms are the candidate MT systems (Aya101, NLLB, Madlad, Gemma2, Llama3.1), the context vector for each source sentence is its LaBSE embedding, and the reward is a normalized blend of BLEU and COMET when references are available, or CometKiwi alone in the target-free case. The four algorithms compared, UCB, LinUCB, Neural LinUCB, and Thompson Sampling, balance exploration and exploitation to identify the best arm, and the claim rests on UCB's regret-minimization behavior surviving on 1,000 samples per language-domain pair.","core_discovery":"The central claim is that model selection for in-domain machine translation can be framed as a contextual multi-armed bandit problem, with each arm being a pretrained NMT system and the context being a LaBSE embedding of the source sentence. The authors show that UCB-driven selection, using a reward that combines BLEU and COMET when gold translations exist or CometKiwi when they do not, identifies the strongest system with only 1,000 exploration sentences and occasionally selects an alternative system that beats the best single model. Averaged over languages and domains, UCB improves on the strongest baseline, NLLB, by 2.68% BLEU, and the paper argues this gives a statistically grounded selection method that avoids fine-tuning and catastrophic forgetting.","pith_inferences":["A direct cross-check would compare UCB selection against the simpler strategy of scoring every model on a small validation set and picking the best once; the paper does not report this baseline, so part of the gain may come from the exploration setup rather than online selection itself.","Because the reported test sentences are used during exploration to compute rewards, the BLEU numbers reflect the bandit's decisions on those same sentences; a strictly separated exploration set and a fresh held-out test set would be a stricter evaluation.","The bandit framing should transfer to other low-resource language families and to other generation tasks with reference-free quality metrics, such as summarization or dialogue response selection.","If the gains survive a stricter held-out evaluation, the approach becomes a practical low-cost alternative to fine-tuning for rapid domain adaptation across many languages."],"forward_implications":["If UCB-based selection works as reported, translators can choose the best system for a domain without fine-tuning, preserving general-domain performance and avoiding the need for large in-domain parallel data.","The target-free result implies that model selection is feasible for languages where only source text exists, using reference-less quality metrics as the reward signal.","The small but consistent gains over the best single model suggest that a pool of diverse pretrained systems can be exploited dynamically, sentence by sentence, rather than committing to one model.","The method offers a statistical basis for selection, reducing the risk of choosing a model from a single high-variance BLEU evaluation on limited data."],"supporting_citations":[{"why":"Supplies the UCB algorithm whose selection behavior yields the reported BLEU gains over the best single model.","marker":"(Auer et al., 2002)"},{"why":"Supplies LinUCB, the contextual bandit algorithm used as one of the comparison methods.","marker":"(Li et al., 2010)"},{"why":"Supplies Neural LinUCB, the deep-representation bandit variant included in the comparison.","marker":"(Xu et al., 2020)"},{"why":"Defines NLLB, the strongest individual model that UCB must beat by an average of 2.68% BLEU.","marker":"(Team, 2022)"},{"why":"Defines Aya101, one of the candidate pretrained translation systems in the arm pool.","marker":"(Üstün et al., 2024)"},{"why":"Provides the Lafand-MT News dataset used for one of the three evaluation domains.","marker":"(Adelani et al., 2022)"},{"why":"Provides the OpenSubtitles Movies dataset used for the conversational domain.","marker":"(Lison and Tiedemann, 2016)"},{"why":"Provides the CCAligned corpus used for the Religious texts domain.","marker":"(El-Kishky et al., 2020)"},{"why":"Defines BLEU, the evaluation metric and part of the reward signal in the parallel-data setting.","marker":"(Papineni et al., 2002)"},{"why":"Supplies CometKiwi, the reference-free reward that enables model selection in the target-free scenario.","marker":"(Rei et al., 2022)"}],"fun_headline_variants":["Bandit beats top African translator by 2.68% BLEU","Multi-armed bandits choose best African NMT, +2.68 BLEU","Bandit model selection lifts African translation 2.68 BLEU","With just 1k sentences, bandit picks best translator, +2.68 BLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage assumes the test set is genuinely held out and was not used for hyperparameter tuning or bandit exploration; if the same 1,000 sentences served both roles, the gains are optimistically biased.","fun_headline_variants_meta":{"raw":{"variants":["Bandit beats top African translator by 2.68% BLEU","Multi-armed bandits choose best African NMT, +2.68 BLEU","Bandit model selection lifts African translation 2.68 BLEU","With just 1k sentences, bandit picks best translator, +2.68 BLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4006,"prompt_tokens":838,"completion_tokens":3168,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":3078}},"tokens_in":454,"tokens_out":3168,"duration_ms":21328,"temperature":1.0,"reasoning_tokens":3078,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:24:35.464612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the procedure with a strictly separated validation set and a fresh 1,000-sentence test set for each language-domain pair, and check whether UCB still beats the best single model by the reported margin with overlapping confidence intervals.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OpenSubtitles Movies dataset used for the conversational domain."},{"cited_title":"CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs","cited_arxiv_id":"1911.06154","evidence_quote":"Provides the CCAligned corpus used for the Religious texts domain."}],"review_version":1}