{"id":"a50dacc6-16e9-489d-8d76-a49aa5d92587","arxiv_id":"2507.21568","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Generating several candidate translations per source sentence for knowledge distillation yields better small multilingual translators than standard single-hypothesis distillation, especially in low-resource settings.","lead":"This paper shows that training a small translation model on multiple different translations of the same sentence, produced by a large teacher model, works better than training on a single best translation. It offers a practical recipe for shrinking huge multilingual translation models into small, deployable students for languages with little data, while also reducing gender bias in the output.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim depends on chrF++ rankings across all seven pairs, but only Swahili pairs are validated with COMET; metric sensitivity is already visible for bam-swh, so a human or neural-metric check is needed before accepting MHD's advantage.","rationale":"The reader identified metric reliability as the weakest assumption, and I agree. The paper makes a strong, general claim about MHD improving student models in low-resource settings, but the evaluation rests on chrF++ alone for seven pairs, with COMET validation only for Swahili. The bam-swh BLEU/chrF++ disagreement for D10_BS shows that metric choice can change conclusions in the extreme low-resource setting. That said, the key comparison (sampling-based MHD vs D1_BS) is consistent across BLEU and chrF++ for all pairs, and the paper includes a useful control experiment (selecting the best translation from D10_top-p for eng-swh) indicating that diversity, not merely the 10x larger corpus, drives the improvement. These strengths mean the concern warrants conditionality, not rejection. A human or SSA-COMET check on the lowest-resource pairs would settle whether the rankings are real. Secondary concerns (no error bars; gender-bias differences of 1-2 points without significance tests; overstatement that 100k MHD matches 1M KD for swh-eng) reinforce CONDITIONAL but are less central. Since my read does not change the reader's verdict, I recommend UNCHANGED.","tokens_in":24469,"tokens_out":11254,"duration_ms":129242,"concrete_test":"Run a pairwise human preference evaluation on a stratified sample of 100 FLORES+ devtest sentences per language pair, comparing student outputs of D10_top-p vs D1_BS for at least ibo-eng, bam-eng, eng-bam, and bam-swh, with two native-speaker annotators per sentence. Compute the human preference ratio and compare with the chrF++ deltas in Table 9. If the human-preferred system differs from the chrF++-preferred system in any pair, the central claim is metric-dependent. A cheaper complementary check: score all systems with SSA-COMET (if it covers these languages) and verify that D10_top-p > D1_BS in every pair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparative claim is that training on M=10 sampled teacher translations (e.g., top-p) improves student models over standard D1_BS sequence-level KD across low-resource directions. All seven language pairs are evaluated primarily with chrF++ on FLORES+ devtest; Section 4.3 states 'we adopt chrF++ as our primary evaluation metric' and validates with COMET only for eng-swh and swh-eng, with no human evaluation reported. For the lowest-resource pairs involving Igbo and Bambara, metric reliability is the load-bearing assumption. Internal evidence already shows metric sensitivity: for bam-swh at 100k, BLEU ranks D10_BS (BLEU 1.2) below D1_BS (BLEU 2.1), while chrF++ ranks D10_BS (chrF++ 20.4) above D1_BS (chrF++ 8.7); the two metrics disagree on whether MHD with beam search helps. Although the key comparison (D10_top-p vs D1_BS) is directionally consistent across BLEU and chrF++ for all pairs, if chrF++ is miscalibrated for Igbo or Bambara (e.g., due to non-standard orthography or morphology), the rank order of sampling-MHD over single-beam KD could be a metric artifact, directly undermining the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Multi-Hypothesis Distillation (MHD), a sequence-level knowledge distillation method that trains a compact student translation model on multiple teacher-generated translations per source sentence. The teacher is an NLLB-200 model and the students are 65M-parameter Transformers. Experiments cover seven low-resource directions (eng-swh, eng-ibo, eng-bam, swh-eng, ibo-eng, bam-eng, bam-swh), two teacher sizes (1.3B and 3.3B), several decoding methods (beam search, diverse beam search, top-p, top-k, MBR), sweeps over the number of hypotheses M, corpus-size sweeps, and analyses of gender bias, hallucinations, vocabulary coverage, and decoding-parameter sensitivity. The central claim is that MHD with M=10 hypotheses, especially with sampling-based decoding, improves student performance over standard single-hypothesis beam-search KD while also reducing gender-bias amplification and hallucinations.","tokens_in":24767,"tokens_out":6144,"duration_ms":73163,"significance":"If the claims hold, the paper makes a useful practical contribution: it shows that a black-box multilingual teacher accessed only through decoding can be distilled into a much smaller bilingual student using monolingual data, and that sampling-based hypothesis generation can beat beam-search KD in low-resource settings. The study is unusually thorough: 482 trained models, two teacher scales, significance testing with paired approximate randomization, and public code. The vocabulary-coverage and corpus-size analyses (Figures 7-9) are informative and could guide practitioners. The claims are falsifiable and the experimental protocol is mostly reproducible from the description.","major_comments":[{"comment":"The comparison between D^10_Z and D^1_BS confounds the number of hypotheses per source with the total number of training examples and optimization steps. A 100k-source D^10_Z corpus contains 1M target sentences, while D^1_BS contains 100k target sentences. The 'best translation per source' control reported in Section 5.1 rules out a single lucky translation but does not control for data quantity. To attribute the gains to diversity rather than to tenfold more training data, please add a control in which the single D^1_BS translation is repeated ten times per source sentence, or otherwise match the total number of target sentences across conditions.","section":"Section 5.1, Eq. (3), Fig. 2"},{"comment":"The zero-shot direction bam-swh shows a clear metric disagreement: BLEU ranks D10_BS below D1_BS (1.2 vs 2.1), while chrF++ ranks it above (20.4 vs 8.7). The conclusion that MHD with beam search helps in the zero-shot scenario therefore rests entirely on chrF++, which is validated with COMET only for eng-swh and swh-eng in Section 4.3. Please add a neural metric or human evaluation for at least bam-swh, or restrict the claim to sampling-based MHD, for which BLEU and chrF++ agree in direction.","section":"Section 5.1, Tables 8-9"},{"comment":"The gender-bias reductions reported in Table 4 are small (e.g., eng-swh D10_BS 51.0 vs D1_BS 49.2; eng-bam D10_BS 50.3 vs 50.8) and no significance testing, confidence intervals, or run-level variance is reported. Since bias mitigation is part of the headline contribution, please provide uncertainty estimates or significance tests for these differences, or soften the corresponding conclusion.","section":"Section 5.4, Table 4"}],"minor_comments":[{"comment":"The caption reads 'ChrfF++ scores'; the metric should be written 'chrF++'.","section":"Table 1 caption"},{"comment":"The text spells 'Kullback-Leiber'; the correct name is 'Kullback-Leibler'.","section":"Section 2.1"},{"comment":"The caption contains an unmatched parenthesis in 'the swh-eng) training corpus'; please fix the parenthetical.","section":"Figure 8 caption"},{"comment":"The captions refer to underlined and bolded values, but these visual markers are not described in the text; please state explicitly in the caption which comparison each marker refers to and ensure the markers are visible in the published PDF.","section":"Appendix D.4, Tables 8-9"},{"comment":"The received/revised dates in the JAIR template ('Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009') appear to be leftover template text and should be updated or removed.","section":"Template front matter"}],"recommendation":"major_revision","confidential_remarks":"The study is substantial and the main sampling-vs-beam result is directionally consistent across BLEU and chrF++ for all pairs, so the core idea is likely sound. The data-quantity confound and the bam-swh metric disagreement are addressable with additional experiments or more careful claims, and I would not reject on the current evidence. The overlap with the authors' NAACL 2025 Findings paper [17] is disclosed, and this version adds enough new material (MBR, hallucinations, word-level KD baseline, corpus-size analysis) to be a distinct contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is real: across seven language pairs and two teacher sizes, a 65M student trained on ten sampled hypotheses (top-p or top-k) consistently beats the standard single-beam sequence-level KD baseline. That holds by chrF++, and for the key comparison it is directionally consistent with BLEU, which matters because the paper only validates chrF++ against COMET for the two Swahili directions. The stress-test worry about Igbo and Bambara metric calibration is legitimate but not load-bearing for the central claim; it could still bite for fine-grained rankings among MBR, top-p, and top-k.\n\nWhat is new is the systematic scope: five decoding methods, an M sweep, corpus-size interaction, a word-level KD baseline, and the bias and hallucination analyses. Prior work [59,17] had the multi-hypothesis idea; this paper generalizes and qualifies it. The MBR and beam-search results, including the degradation of D10_BS for bam-swh, are informative. The corpus analysis (Zipf, self-BLEU, vocabulary coverage) is a real strength, and the code and data are public.\n\nSoft spots, in order. First, the abstract and conclusion overstate the corpus-size benefit: Figure 7 and Table 9 show that 100k MHD does not match 1M standard KD for swh-eng, so the claim of achieving similar results with a much smaller corpus needs a qualifier. Second, the hallucination-reduction claim rests on visual overlap of KDEs, without a quantitative cutoff or significance test; it is plausible and consistent with the bias numbers, but it is weaker evidence than the translation-quality results. Third, metric validation is thin for three of seven pairs. Given the bam-swh BLEU/chrF++ disagreement on D10_BS versus D1_BS, adding a neural metric or a small human check for Bambara and Igbo would harden the paper. None of these kill the central claim; they are revision-level issues.\n\nThis paper is for practitioners who want a cheap recipe for distilling NLLB-class teachers into deployable students for low-resource directions, and for researchers working on sequence-level KD. It deserves peer review: the experiments are extensive and reproducible enough that referees can check the central numbers quickly. I would send it to review with a request to fix the overstatement and make the hallucination analysis quantitative, but I would not block acceptance on the metric concern given the BLEU consistency for the main comparison.","headline":"A careful, reproducible empirical study showing that training students on multiple sampled teacher translations beats single-beam KD for low-resource pairs; the main claims hold, though the abstract oversells the corpus-size result and the hallucination analysis is thin.","tokens_in":25346,"tokens_out":2063,"would_cite":true,"duration_ms":25862,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a small translation model on several translations of each source sentence, rather than the teacher's single best beam-search output, improves quality for low-resource languages and also softens two known side effects of…","keywords":["multi-hypothesis distillation","sequence-level knowledge distillation","low-resource machine translation","decoding methods","multilingual neural machine translation","gender bias","hallucinations","chrF++"],"falsifier":"Have human translators rank the FLORES+ devtest outputs of the D1-BS, D10-top-p, and D10-MBR students for eng-ibo and bam-swh; if the sampling- or MBR-trained students do not beat the beam-KD student, the central claim fails.","tokens_in":24284,"feed_emoji":"🌍","tokens_out":10501,"duration_ms":109245,"temperature":0.7,"pith_summary":"Large multilingual translation models are too big for many real uses, so they are often compacted by knowledge distillation: a small student is trained on translations that a large teacher produces. The standard recipe keeps only the teacher's single most-likely beam-search translation per sentence. This paper argues that this discards most of what the teacher knows and proposes Multi-Hypothesis Distillation (MHD), which generates several translations per source sentence and trains the student on all of them. Across seven low-resource directions involving Swahili, Igbo, and Bambara, MHD students match or beat the standard distilled students, even when the added translations are individually lower quality than the beam output. The same diversity also reduces the gender-bias amplification and the hallucinations that distillation typically brings.","feed_headline":"Ten sampled translations beat one beam translation","feed_subtitle":"For low-resource pairs, training a student on diverse teacher outputs improves quality and cuts gender bias.","key_machinery":"The central object is the teacher's output distribution, represented in MHD by $M$ decoded hypotheses per source instead of a single mode. The training signal is $$\\mathcal{L}_{\\mathrm{MHD}} = -\\sum_{i=1}^{N}\\sum_{m=1}^{M}\\sum_{t=1}^{T_i} \\log P(\\tilde{y}_{i,m,t} \\mid \\tilde{y}_{i,m,<t}, x_i; \\theta_S),$$ which is simply the standard sequence-level KD loss on a corpus where each source sentence appears $M$ times. The decoding method is what shapes that corpus: beam search and diverse beam search return ranked high-probability lists, giving low variability and, for poorly fitted languages, increasingly improbable continuations as $M$ grows; top-$p$ and top-$k$ sample independently, giving high lexical variability and stable probabilities; MBR reranks epsilon-sampled candidates by expected chrF, filtering out bad translations. The paper uses these properties to explain when MHD helps: sampling wins where the teacher is weak or the corpus is small, while high-quality ranked outputs regain the edge when monolingual data are abundant.","core_discovery":"MHD is a sequence-level knowledge distillation method: a teacher translation model (NLLB-200 in the 1.3B and 3.3B sizes) decodes $M$ hypotheses $\\tilde{y}_{i,1}, \\ldots, \\tilde{y}_{i,M}$ for each source sentence $x_i$ using one of five decoding strategies, and the student is trained with the standard cross-entropy objective on the repeated dataset, so every source appears $M$ times paired with a different target. The paper's central empirical claim is that, for low-resource directions, this beats the standard sequence-level KD baseline $D^1_{BS}$ in which each source has only its beam-search output. Gains are largest when the hypotheses are sampled (top-$p$ and top-$k$) rather than ranked (beam search and diverse beam search), with MBR decoding best for the least-resourced Bambara pairs. The paper also reports that multi-hypothesis training reduces gender-bias amplification as measured by contrastive conditioning, and reduces hallucinations in most settings, although multiple beam-search hypotheses from a weak teacher can increase them for Bambara.","pith_inferences":["Beyond the paper: if the gains come from diversity rather than from the identity of any single good translation, then controlling diversity directly, for example by tuning top-$p$ while filtering repeated sentences, could further improve student models at lower cost than MBR.","Beyond the paper: the paper's vocabulary-coverage curves suggest a practical recipe: with scarce monolingual data, spend distillation effort on covering the target vocabulary with several sampled hypotheses first, then switch to high-quality single hypotheses once coverage saturates.","Beyond the paper: because MHD needs only monolingual text and access to the teacher's outputs, the same recipe could be applied to even lower-resource languages or to API-only teachers, and could be combined with back-translation to grow the source side as well.","Beyond the paper: the multi-hypothesis idea is not specific to translation; other sequence-generation tasks where the teacher's mode is unrepresentative might benefit from the same keep-several-outputs training signal, but that is an extrapolation."],"forward_implications":["MHD reaches results comparable to standard sequence-level KD while using a much smaller monolingual corpus, so the method lowers the data requirement for distilling a competitive student.","For the lowest-resource directions (involving Bambara and the into-English pairs), sampling-based hypotheses give the strongest students; top-$p$ is nearly as effective as MBR and much faster.","With one million source sentences, the advantage of sampling over beam search narrows, and diverse beam search can even lead for Swahili-English; the right decoding choice depends on corpus size and teacher quality.","Where the teacher is poorly calibrated, multiple beam-search hypotheses degrade student quality, while multiple sampled hypotheses keep improving it, which is direct evidence that the mode is not a good summary of the teacher's distribution.","Training on $M=10$ hypotheses systematically mitigates gender-bias amplification compared to a single beam hypothesis, with sampling methods reducing it most."],"supporting_citations":[{"why":"Supplies the NLLB-200 multilingual teacher models and the FLORES+ dev and devtest sets used throughout the experiments.","marker":"[40]"},{"why":"Defines sequence-level knowledge distillation, the baseline framework MHD modifies by replacing the single synthetic target with M targets.","marker":"[28]"},{"why":"Argues that the mode of the probability distribution is not a reliable summary of it, the theoretical basis for using multiple hypotheses.","marker":"[11]"},{"why":"Defines word-level KD, whose distributional knowledge motivates sampling a broader range of the teacher's distribution in MHD.","marker":"[25]"},{"why":"Defines top-p (nucleus) sampling, one of the sampling decoding methods shown to produce the most diverse corpora.","marker":"[26]"},{"why":"Defines Minimum Bayes Risk decoding, the method that produces the best students for the extremely low-resource Bambara directions.","marker":"[36]"},{"why":"Introduces contrastive conditioning, the protocol used to measure gender-bias amplification in the student models.","marker":"[54]"},{"why":"Documents that knowledge distillation amplifies gender bias, the failure mode MHD is shown to reduce.","marker":"[3]"},{"why":"Shows that sentence-embedding similarity detects machine-translation hallucinations, the method used in the hallucination analysis.","marker":"[6]"},{"why":"Supplies the paired approximate randomization test used to decide whether differences between student models are statistically significant.","marker":"[45]"}],"fun_headline_variants":["Sampling beats beam for low-resource translation distillation","Multiple teacher translations beat one beam output","Diverse teacher hypotheses cut bias and improve translation","Multi-hypothesis distillation boosts low-resource translation quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that chrF++ on the FLORES+ devtest set judges translation quality faithfully for all seven language pairs; the paper validates against COMET only for English-Swahili and Swahili-English, so a metric artifact for Igbo or Bambara would undermine the ranking at the center of the claim.","fun_headline_variants_meta":{"raw":{"variants":["Sampling beats beam for low-resource translation distillation","Multiple teacher translations beat one beam output","Diverse teacher hypotheses cut bias and improve translation","Multi-hypothesis distillation boosts low-resource translation quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001206,"raw_usage":{"total_tokens":4963,"prompt_tokens":935,"completion_tokens":4028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":3968}},"tokens_in":551,"tokens_out":4028,"duration_ms":27843,"temperature":1.0,"reasoning_tokens":3968,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:36:56.301100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human translators rank the FLORES+ devtest outputs of the D1-BS, D10-top-p, and D10-MBR students for eng-ibo and bam-swh; if the sampling- or MBR-trained students do not beat the beam-KD student, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Minimum Bayes Risk decoding, the method that produces the best students for the extremely low-resource Bambara directions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the paired approximate randomization test used to decide whether differences between student models are statistically significant."}],"review_version":1}