{"id":"0cfa6136-666b-4a7b-bf87-3adeca972ac4","arxiv_id":"2411.16300","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Cross-lingual translation instruction tuning transfers capabilities from Chinese and English to 100+ low-resource languages, with measured gains on translation and understanding benchmarks.","lead":"BayLing 2 fine-tunes Llama models on 3.2 million instructions that mix Chinese and English with translation tasks into 100+ languages, aiming to transfer language abilities from high- to low-resource languages. The paper reports improved translation and understanding scores on several low-resource languages while keeping high-resource performance roughly intact.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Potential train/eval overlap on Flores/FLORES-based benchmarks could make the reported multilingual capability transfer an artifact of memorization; the paper never states that Flores-derived sentences were excluded from training.","rationale":"The reader's weakest assumption correctly targets the unstated condition that evaluation data is disjoint from training data. This is the most load-bearing assumption because the paper's headline contribution is an efficient mechanism for transferring capabilities to low-resource languages, and the primary evidence is performance on Flores-101 and Belebele. Both benchmarks derive from the Flores sentence pool, and the paper gives no explicit statement that these sentences were filtered out during dataset construction. The training data description is too vague to rule out accidental inclusion, and the dataset is not released. The concern is not that the authors acted dishonestly; it is that the experimental design as described does not currently exclude a plausible memorization-based alternative. If the overlap check comes back clean, the central claim remains credible and the paper still merits conditional acceptance with the dataset released for verification. If overlap is found, the central claim would need to be substantially revised. I therefore agree with the reader's conditional verdict rather than moving to reject, because the issue is verifiable and the rest of the evidence (WMT22 results on more recent test data, and the ablation showing cross-lingual instructions matter) provides partial support for the method. The concrete test is the minimal check that would settle whether the concern actually lands.","tokens_in":27421,"tokens_out":3512,"duration_ms":36302,"concrete_test":"Release the training corpus or a searchable hash of it and run exact-match plus normalized 8-gram containment checks between the training set and Flores-101 dev/test, FLORES-200/Belebele dev/test, and WMT22 test sets. Report overlap rates per language. If any overlap is found, re-evaluate BayLing-2-8B on a freshly collected parallel corpus and on Belebele passages confirmed absent from training; if the gains disappear, the capability-transfer claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that fine-tuning on cross-lingual translation instructions alone transfers knowledge and understanding to low-resource languages. That conclusion requires the evaluation benchmarks to be disjoint from the training corpus. Section 3.1 describes the 3.2M-instruction corpus only as 'primarily sourced from news corpora' and does not mention any deduplication against benchmark sets. Flores-101 is itself a news/travel/books parallel corpus, and Belebele passages are explicitly built from FLORES-200, which shares its sentence pool with Flores-101. If Flores sentences appeared in the translation training pairs, then both §4.1.1 translation scores and §4.1.2 Belebele comprehension scores could be inflated by memorization or pattern matching rather than by genuine language alignment. The reported Belebele gains are small (e.g., BayLing-2-8B vs Llama-3-8B-Instruct: 0.3189 vs 0.3033 for Bambara, 0.3378 vs 0.3244 for Zulu) and close to the 0.25 chance floor, so even modest overlap could account for the observed improvements. The paper also does not release the training set, making this overlap impossible to rule out from the text alone. If overlap is present, the paper's main evidence for knowledge transfer collapses; if absent, the concern is resolved but should be stated explicitly for reproducibility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents BayLing 2, a family of multilingual LLMs (BayLing-2-7B, BayLing-2-13B, BayLing-2-8B) obtained by instruction-tuning Llama models on a 3.2M-instruction corpus consisting of Chinese and English instructions plus cross-lingual translation instructions between those two pivot languages and 100+ other languages. The central claim is that this \"language alignment\" transfers knowledge, generative capability, and instruction-following ability from high-resource to low-resource languages without constructing native instruction data for each language. Evaluation covers translation (Flores-101, WMT22), multilingual understanding (Belebele, Multilingual HellaSwag, XNLI, Multilingual ARC), and English/Chinese general benchmarks, with an ablation that removes cross-lingual instructions.","tokens_in":27650,"tokens_out":5757,"duration_ms":51501,"significance":"If the central claim holds, the paper offers a practical and cost-effective recipe: translation-style parallel instruction data can improve a large number of low-resource languages, avoiding the prohibitive cost of annotating native instruction data for each language. The paper's strengths are its breadth—three model scales, 100+ languages, four multilingual understanding benchmarks, and comparisons against strong open baselines—and the public release of models and code. However, the evidence for the headline knowledge-transfer claim currently depends on an unverified assumption that training data are disjoint from the evaluation benchmarks, and on point estimates without uncertainty quantification; both of these gaps are load-bearing for the claimed transfer result.","major_comments":[{"comment":"The training-data description never states that Flores-101/FLORES-200-derived sentences were excluded from the 3.2M instructions; it only says the instructions are \"primarily sourced from news corpora.\" Since Flores-101 sentences come from news, travel guides, and books, and Belebele passages are explicitly built from FLORES-200, any inclusion of such sentences in the translation training pairs would inflate both the translation scores in §4.1.1 and the Belebele comprehension gains in §4.1.2. The latter is the main non-translation evidence for capability transfer, so the authors should state whether Flores/FLORES sentences were removed from training, report overlap statistics against the evaluation sets, or release the training corpus; without this, the central claim cannot be distinguished from memorization.","section":"§3.1 (dataset construction; used in §4.1.1 and §4.1.2)"},{"comment":"The phrase \"significant improvements across over 20 low-resource languages\" is not supported by any significance test, error bar, or standard deviation; all reported numbers are point estimates from single evaluations, and Figure 8 shows only a selected subset of languages with no aggregate count or list. Several differences are near the noise floor (e.g., XNLI Urdu: 0.3454 vs 0.3454; Multilingual HellaSwag Vietnamese: 0.4025 vs 0.4021; Belebele Zulu: 0.3378 vs 0.3244), and on XNLI Bulgarian, BayLing-2-8B is worse than Llama-3-8B-Instruct (0.4510 vs 0.4518). The authors should provide variance or significance estimates, or soften the claim to a descriptive statement of observed improvements.","section":"§4.1.2, Figure 8, Tables 9–12"},{"comment":"The ablation study is presented only graphically, with no numerical scores for the \"w/o cross-lingual instructions\" variant in the main text or the appendix. The claims that removing cross-lingual instructions causes a \"significant performance decline\" in low-resource languages and that the remaining variant suffers from inter-language conflicts are therefore not checkable from the paper. Exact numbers for both variants on Belebele and on the Chinese/English general benchmarks should be included in a table.","section":"§4.3, Figure 10 and Figure 11"},{"comment":"The translation gains in §4.1.1 are partly by construction because the model is fine-tuned on translation instructions and evaluated on translation tasks; they cannot by themselves establish the paper's broader claim of knowledge transfer to low-resource languages. The Belebele and multilingual-understanding results carry that weight, but they are currently under-quantified and potentially contaminated by train/eval overlap (as noted in the first major comment). The paper should make this reasoning explicit and present the required safeguards—benchmark disjointness and uncertainty quantification—together with the transfer claim.","section":"§4.1.1 (Table 1) and §4.1.2 (p. 9)"}],"minor_comments":[{"comment":"The caption says \"WMT22 benchmark\" but the table reports averages over roughly 100 Flores-101 languages; this appears to be a copy-paste error and should be corrected.","section":"Table 1"},{"comment":"The header \"BELU\" in Table 2 and \"Mulitlingual preformance\" in Table 1 should be corrected to \"BLEU\" and \"Multilingual performance.\"","section":"Tables 1 and 2"},{"comment":"The statement that BayLing's training data \"does not include instruction data for these low-resource languages but only cross-lingual instructions\" is misleading because cross-lingual instructions contain low-resource-language source or target text; it should be rephrased to say the corpus contains no monolingual instruction-following data in those languages.","section":"§4.1.2, p. 9"},{"comment":"The benchmark list uses \"ammlu\" and \"aexams\" without defining them in the text; these abbreviations should be expanded in the benchmark descriptions or table caption.","section":"§4.2"},{"comment":"GSM8K is described as being described in section 4.1.2, but it is first presented in §4.2; the cross-reference is incorrect.","section":"§4.2"},{"comment":"These figures are visually dense and the legends are difficult to read; since they carry the ablation claim, a table of the underlying scores would be clearer and more useful to readers.","section":"Figures 10 and 11"},{"comment":"The text uses both \"Swati\" and \"Swazi\" for the same Belebele language subset; a single consistent label should be used throughout.","section":"§4.1.2, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The train/eval overlap concern is the most consequential issue: if the authors can supply a clear deduplication statement and an overlap analysis, the paper could become acceptable. The scope of the \"knowledge transfer\" claim should also be revisited, since Belebele and the selected multilingual HellaSwag/XNLI/ARC subsets are narrower than full knowledge benchmarks, and the paper does not state whether the 3.2M-instruction training corpus will be released despite the emphasis on reproducibility and openness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BayLing 2 is a practical engineering contribution: a 3.2M-instruction corpus mixing Chinese/English instructions with cross-lingual translation instructions for 100+ languages, and a set of fine-tuned Llama models. The negative-result ablation — removing cross-lingual instructions hurts low-resource performance — is the most interesting piece, and the WMT22 translation numbers are genuinely strong for open models of this size. The authors also release models and code, which helps reproducibility.\n\nThe soft spot the stress-test note identified is real and not minor. Belebele is built directly from FLORES-200 passages, and Flores-101 shares that sentence pool. Section 3.1 says the instruction data was 'primarily sourced from news corpora' but never says FLORES sentences were deduplicated or excluded. Since the training set is not released, overlap cannot be checked. If Flores sentences appeared in the translation pairs, both the Flores translation scores and the Belebele comprehension gains — which are small and close to the 0.25 chance floor — could be inflated by memorization rather than by capability transfer. The WMT22 results are less exposed, but the central claim that translation data transfers knowledge to low-resource languages rests mainly on Belebele, so this matters.\n\nOther gaps are more moderate: no error bars or significance tests, an ablation reported only graphically, and some inconsistent translations where BayLing-2-8B loses to Llama-3-8B-Instruct on Flores directions. That doesn't doom the project — translation directions are noisy — but the selective presentation makes the 'significant improvements' claim hard to credit.\n\nOverall, the method is plausible and probably useful: translation alignment at this scale is an efficient way to expand language coverage, and the inter-language conflict finding is a worthwhile observation. The paper deserves a serious referee, but the authors should have to release the dataset or explicitly document deduplication against FLORES/Belebele, report error bars, and show full results rather than cherry-picked subsets. If overlap is confirmed, the capability-transfer claim mostly collapses; if it isn't, the claim is credible. That is the right question to send back on.","headline":"Useful multilingual alignment recipe at scale, but the paper must address FLORES/Belebele overlap before the capability-transfer claim is credible.","tokens_in":28201,"tokens_out":2476,"would_cite":false,"duration_ms":39054,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BayLing 2's central claim is that fine-tuning an LLM on Chinese/English instructions plus cross-lingual translation tasks transfers knowledge and generative ability to 20+ low-resource languages.","keywords":["multilingual large language models","cross-lingual instruction tuning","language alignment","low-resource languages","machine translation","knowledge transfer","multilingual evaluation","instruction following"],"falsifier":"Take the reported low-resource test languages such as Bambara, Luganda, Swahili, Zulu, and Lao and build a fresh parallel and comprehension benchmark from sources not derived from the Flores corpus; if BayLing-2's advantage over the base Llama models shrinks to zero or reverses on that held-out data, the capability-transfer claim is falsified. A cheaper first check is to run a near-duplicate membership test between the 3.2-million-instruction dataset and the Flores-101/Belebele test sets.","tokens_in":27203,"feed_emoji":"🌐","tokens_out":19401,"duration_ms":146398,"temperature":0.7,"pith_summary":"BayLing 2 is an attempt to make large language models useful in many languages without collecting native instruction data for each one. The paper's central claim is that instruction-tuning on a corpus of 3.2 million prompts, dominated by Chinese and English instructions plus translation-style cross-lingual instructions linking those pivot languages to over 100 other languages, transfers knowledge, generative ability, and instruction-following from high-resource to low-resource languages. On the Flores-101 and WMT22 translation benchmarks the resulting models outperform open-source models of similar scale, and on Belebele, Multilingual HellaSwag, XNLI, and Multilingual ARC they show gains in more than 20 low-resource languages such as Bambara, Luganda, Swahili, and Zulu. The paper also argues that the cross-lingual instructions prevent the inter-language conflict that appears when fine-tuning only on high-resource-language instructions, and concludes that translation data, which is far cheaper to obtain than hand-written instructions in every language, can serve as an efficient alignment mechanism for multilingual capability transfer.","feed_headline":"Training on translations lifts LLMs in 20+ low-resource languages","feed_subtitle":"BayLing 2 shows that cross-lingual translation data can transfer LLM capabilities to low-resource languages.","key_machinery":"The load-bearing mechanism is the cross-lingual instruction, defined as a prompt that asks the model to transform or continue the same content across languages, chiefly through interactive translation, constrained translation, document-level translation, and single-sentence translation between Chinese/English and one of 100+ other languages. These instructions act as a bridge that ties the target language's surface form to the semantic space already occupied by the high-resource pivot languages, so knowledge and generation routines learned in Chinese and English can be reused in low-resource languages. The instruction dataset, 3.2 million instructions totaling about 1.47 billion tokens, deliberately mixes high-resource instructions with cross-lingual ones; the cross-lingual portion does the alignment work, while the high-resource portion preserves general instruction-following. Rendering the same sentence in multiple languages around a pivot, as in the paper's speed-of-light example, makes the semantic equivalence explicit to the model.","core_discovery":"On its own terms, the paper's discovery is that a model does not need instruction data written in a target language to become capable in that language; it needs only a bridge. By fine-tuning Llama-based models on high-resource instructions (Chinese and English) interleaved with cross-lingual instructions that express the same content in another language, BayLing 2 aligns the representations of low-resource languages with the high-resource languages where the model already has strong knowledge and generation. The paper reports measurable improvements: for translation, consistent quality gains across most of the 101 Flores languages and top open-source results on WMT22; for knowledge and understanding, improvements across more than 20 low-resource languages on reading comprehension, commonsense inference, natural-language inference, and science reasoning. An ablation that removes all cross-lingual instructions shows both a drop in low-resource performance and a decline in Chinese benchmarks, which the paper attributes to inter-language conflict; reintroducing the cross-lingual bridge repairs both. The paper's conclusion is that language alignment through cross-lingual instruction tuning is an efficient route to multilingual LLMs and that it does not sacrifice high-resource-language performance.","pith_inferences":["Editorial inference: the same recipe should work with pivot languages other than Chinese and English; any high-capability language with abundant parallel text could act as the bridge, making the method a general recipe rather than a Chinese/English-specific result.","Editorial inference: because the ablation only removes the cross-lingual instructions as a whole, it does not isolate whether the gains come from the translation objective itself or simply from exposing the model to more multilingual text; a control fine-tuned on matched monolingual low-resource text would separate these explanations.","Editorial inference: the alignment account predicts a dose-response relationship, namely that languages with more cross-lingual training sentences should show larger gains; plotting per-language improvement against the number of training instructions per language would test that prediction directly.","Editorial inference: the knowledge-transfer claim is currently demonstrated on multiple-choice benchmarks; evaluating BayLing-2 on open-ended generation tasks in languages like Bambara, Luganda, or Zulu would show whether the alignment transfers generative fluency as well as answer selection."],"forward_implications":["Low-resource language capability can be improved with translation data rather than hand-built instruction data for each language, since parallel text is abundant and cheaper to collect.","Supervised fine-tuning for multilingual use should include cross-lingual instructions, not just target-language instructions; without them the model suffers inter-language conflict and loses low-resource and even Chinese performance.","Off-target generation in low-resource languages, where the base model emits the wrong language or script, is reduced by alignment, making the model usable for translation into those languages.","Models trained this way keep pace with or exceed other open-source models of the same size on translation and come close to closed-source systems on WMT22 high-resource pairs.","Improvements on knowledge and reasoning benchmarks in 20+ low-resource languages imply that the transferred capability is not limited to translation itself but includes understanding and knowledge answering."],"supporting_citations":[{"why":"Supplies the cross-lingual alignment method and interactive-translation recipe that BayLing 2 scales from a Chinese-English pair to 100+ languages.","marker":"[Zhang et al., 2023]"},{"why":"Provides the Llama foundation models and documents the language imbalance in pre-training corpora that motivates the transfer approach.","marker":"[Touvron et al., 2023]"},{"why":"Defines the Belebele benchmark built on FLORES-200, the main source of the paper's low-resource reading-comprehension results.","marker":"[Bandarkar et al., 2023]"},{"why":"Supplies Multilingual HellaSwag and Multilingual ARC, the benchmarks used to measure commonsense and science-reasoning transfer.","marker":"[Dac Lai et al., 2023]"},{"why":"Supplies XNLI, the cross-lingual natural-language-inference benchmark used to measure understanding transfer.","marker":"[Conneau et al., 2018]"},{"why":"Defines the BLEU scoring procedure (sacrebleu) used for translation evaluation.","marker":"[Post, 2018]"},{"why":"Defines COMET, the semantic-similarity metric used alongside BLEU for translation evaluation.","marker":"[Rei et al., 2022]"},{"why":"Provides the evaluation harness under which all benchmark results are computed, making scores comparable across models.","marker":"[Gao et al., 2023]"}],"fun_headline_variants":["Cross-lingual bridge lifts LLMs in 20+ low-resource languages","BayLing 2 aligns LLMs across 100+ languages without target data","Efficient language alignment: BayLing 2 teaches LLMs new tongues from a bridge","BayLing 2: One cross-lingual dataset boosts 100+ language LLMs","Align once, serve many: BayLing 2's efficient multilingual path"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation benchmarks are assumed to contain no sentences that BayLing 2 saw during training; the paper does not state that Flores-101 or FLORES-200-derived data was excluded from the 3.2-million-instruction corpus, so if overlap exists, the reported low-resource gains could partly reflect memorization.","fun_headline_variants_meta":{"raw":{"variants":["Cross-lingual bridge lifts LLMs in 20+ low-resource languages","BayLing 2 aligns LLMs across 100+ languages without target data","Efficient language alignment: BayLing 2 teaches LLMs new tongues from a bridge","BayLing 2: One cross-lingual dataset boosts 100+ language LLMs","Align once, serve many: BayLing 2's efficient multilingual path"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000434,"raw_usage":{"total_tokens":2289,"prompt_tokens":1102,"completion_tokens":1187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":718,"completion_tokens_details":{"reasoning_tokens":1077}},"tokens_in":718,"tokens_out":1187,"duration_ms":14930,"temperature":1.0,"reasoning_tokens":1077,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:15:27.624913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the reported low-resource test languages such as Bambara, Luganda, Swahili, Zulu, and Lao and build a fresh parallel and comprehension benchmark from sources not derived from the Flores corpus; if BayLing-2's advantage over the base Llama models shrinks to zero or reverses on that held-out data, the capability-transfer claim is falsified. A cheaper first check is to run a near-duplicate membership test between the 3.2-million-instruction dataset and the Flores-101/Belebele test sets.","supporting_citations":[{"cited_title":"Xnli: Evaluating cross-lingual sentence representations","cited_arxiv_id":null,"evidence_quote":"Supplies XNLI, the cross-lingual natural-language-inference benchmark used to measure understanding transfer."}],"review_version":1}