{"id":"95c6879b-a833-4195-82c9-e35ae16d6b03","arxiv_id":"2506.13044","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding parallel data during continued pretraining improves a 1.1B LLM's translation and multilingual common-sense reasoning, with end-of-training placement performing best.","lead":"Large language models trained mostly on English gain multilingual skills when parallel translated sentence pairs are added to their training data. This paper shows placement matters: adding the parallel data at the end of training gives the largest translation and multilingual reasoning gains.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Checkpoint selection bias, not just seed variance, undermines the ranking of data-placement strategies.","rationale":"The paper is a well-controlled study with public code and checkpoints, and the large gains from parallel data over no-parallel and monolingual settings are probably robust. However, the headline 'end is most effective' is the paper's most actionable and novel claim, and it is exactly the part that the uneven checkpoint density threatens. The reader's seed-variance concern is real but applies globally and is expensive to test; the checkpoint-selection asymmetry is a deterministic design choice that can be checked without new training runs by reusing saved checkpoints. If the ranking survives uniform checkpoint selection, the paper becomes much stronger. If not, the conclusion should be softened to 'parallel data helps, and end-placement can be effective when checkpointed well,' which is a different message. Thus I recommend keeping the CONDITIONAL verdict, unchanged from the reader. I partially agree with the reader: the identified weakest assumption (single-run variance) is a valid concern, but the more specific, more load-bearing hole is checkpoint selection.","tokens_in":17177,"tokens_out":10675,"duration_ms":99928,"concrete_test":"Recompute Table 3 using a uniform checkpoint selection rule: for each of the seven settings, evaluate the final checkpoint (or the best checkpoint from the same number of equally spaced candidates across the entire run). If PARALLELLAST still outperforms PARALLELDISTRIBUTED by comparable margins on all four language pairs, the ranking holds; if the gap narrows below significance or flips on some pairs, the 'most effective' claim is an artifact of checkpoint density. A cheaper alternative: re-run only PARALLELDISTRIBUTED with checkpoints every 1k steps in its final 5k-step window and apply the same best-dev-BLEU selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that parallel data is most effective when added at the end rests on a comparison of seven settings in Table 3. Section 4 states that checkpoints are saved every 5,000 steps, but 'for the first 5,000 steps of the PARALLELFIRST setting and the last 5,000 steps of the PARALLELLAST settings, we save checkpoints every 1,000 steps.' After training, the checkpoint with the highest average dev BLEU is selected for every setting. Thus the reported translation scores for PARALLELLAST (and PARALLELFIRST) are best-of-five checkpoints over a 5k-step window, while the other settings are best-of-one at each 5k boundary. This systematic asymmetry inflates the apparent advantage of the recommended PARALLELLAST strategy, which is exactly the setting whose ranking the conclusion emphasizes. The margins between PARALLELLAST and PARALLELDISTRIBUTED in zero-shot range from about 4 to 14 BLEU points; if the distributed model had the same checkpoint density, its best checkpoint could be substantially higher. The control is also not exact: the paper acknowledges in footnote 4 that PARALLELLAST trains on 166B or 164B tokens instead of 167B, contradicting the 'consistent training set size and order' claim. This does not refute the qualitative benefit of parallel data, but it does undermine the specific 'end is most effective' ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled study of how parallel sentence pairs affect the multilingual capabilities of decoder-only LLMs. Using a 1.1B TinyLlama model trained on about 167B tokens, it compares seven data-placement strategies: no parallel data, multilingual monolingual data, non-adjacent parallel data, parallel data at the start, parallel data distributed throughout, parallel data at the end in all directions, and parallel data at the end in one direction. Translation is evaluated on WMT-23 and Flores-200 devtest, and common-sense reasoning on English, Chinese, and Indonesian benchmarks. The paper claims that parallel data improves translation and non-English reasoning, that placing parallel data at the end is the most effective placement, that unidirectional parallel training does not transfer to the reverse direction, and that the amount of incidental bilingual signal correlates with translation quality.","tokens_in":17535,"tokens_out":5142,"duration_ms":49311,"significance":"If the ranking of data-placement strategies holds, the paper gives concrete and actionable guidance for multilingual LLM training: preserve the parallel format, add parallel data at the end, and consider unidirectional specialization for translation systems. The release of code, checkpoints, and data, together with evaluation on external test sets, is a genuine strength. The reverse-direction failure of unidirectional models is a striking and falsifiable finding. However, the central placement claim is weakened by an asymmetric checkpoint-selection procedure and by single-run training without seed variation, so the significance is contingent on additional controlled evidence.","major_comments":[{"comment":"The checkpoint-selection procedure systematically favors PARALLELLAST and PARALLELFIRST. The paper states that checkpoints are saved every 5,000 steps, but for the first 5,000 steps of PARALLELFIRST and the last 5,000 steps of PARALLELLAST settings, checkpoints are saved every 1,000 steps, and then the checkpoint with the highest average dev BLEU is selected for every setting. Thus PARALLELLAST and PARALLELFIRST are best-of-five selections over a 5k-step window, while the other settings are best-of-one at 5k boundaries. This asymmetry inflates the apparent advantage of the recommended PARALLELLAST strategy. In Table 3, the zero-shot margins between PARALLELLAST(ALL) and PARALLELDISTRIBUTED range from about 3 to 14 BLEU points (e.g., 35.91 vs. 21.95 for EN→ID and 10.73 vs. 7.40 for ZH→EN), so the conclusion that 'training at the end is the most effective approach' is not supported as a controlled comparison. The authors should evaluate all settings with the same checkpoint density, or compare final checkpoints rather than best-of-variable checkpoints.","section":"Section 4, checkpoint selection"},{"comment":"The token-count control is not exact. Section 3 says 'we maintain the order and quantity of training data across all experiments' and 'preserve a consistent training set size and order,' but footnote 4 reports that PARALLELLAST(ALL) is trained on 166B tokens and PARALLELLAST(UNI) on 164B tokens instead of 167B, because not all parallel data are consumed. This means the PARALLELLAST treatments differ from the NO PARALLEL baseline not only in data placement but also in total token count. The qualitative benefit of parallel data may survive this confound, but the precise ranking in Table 3 and the claim of a consistent control do not.","section":"Section 3 vs. footnote 4"},{"comment":"Each experimental setting is trained only once, so the paired approximate randomization test in Section 5 does not bound training stochasticity. That test compares two single models on test sentences; it cannot tell whether the difference between, say, PARALLELDISTRIBUTED and PARALLELLAST(ALL) would persist under a different random seed or data-order shuffle. Larger, non-overlapping runs are needed for the central ranking, or the paper should present its conclusions as single-run observations and soften the comparative claims accordingly.","section":"Section 4 and Section 5, statistical significance"},{"comment":"The word-level analysis is circular. The paper builds a word translation dictionary by aligning source and target sentences from the WMT-2022 test set (Chinese-English) and the Flores-200 dev set (Indonesian-English), then uses that same dictionary and those same references to judge whether the model's output contains the correct translation. A word's 'correct translation ratio' is therefore partly determined by the dictionary construction procedure on the evaluation sentences themselves. To support the claim that incidental bilingual signal frequency affects translation capability, the dictionary should be induced from a separate held-out parallel corpus and then applied to a disjoint test set.","section":"Section 6.4, word-level bilingual-signal analysis"}],"minor_comments":[{"comment":"The CometKiwi thresholds (0.42 for Chinese-English and 0.58 for Indonesian-English) are described as set 'based on manual observation'; the paper should state how many examples were inspected and whether the thresholds were fixed before or after seeing the translation results, since threshold choice can affect the filtered-data comparison.","section":"Section 6.2, threshold selection"},{"comment":"In Table 3, PARALLELLAST(UNI) is shown as one row, but it represents four separately trained unidirectional models; the table should make explicit that each cell in that row comes from a different model.","section":"Table 3 and Table 5, presentation of PARALLELLAST(UNI)"},{"comment":"Figure 2 should state whether the BLEU curve is computed on the development set or the test set, and should clearly identify which curve corresponds to NOPARALLEL and which to PARALLELFIRST.","section":"Figure 2"},{"comment":"Using the WMT-2022 test set and Flores-200 dev set as the checkpoint-selection development set is unusual and could confuse readers; consider renaming them 'selection sets' or clearly stating that no held-out test data are used for selection.","section":"Section 4, development set terminology"},{"comment":"The decline in English common-sense accuracy for PARALLELLAST(ALL) (50.44 for NO PARALLEL to 49.08 for PARALLELLAST(ALL) in Table 4) is reported descriptively; a discussion of whether this is within expected single-run variation would be useful given that only one run is available.","section":"Section 5, English common-sense reasoning"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-scoped empirical study with useful public artifacts, but the central placement claim rests on a checkpoint-selection asymmetry and single-run comparisons. If the authors can supply additional runs or a re-analysis with uniform checkpoint selection, the paper would be substantially stronger. The citation pattern appears appropriate; no concerns about novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper up front: it's a genuinely controlled comparison of how parallel data placement affects a 1.1B decoder-only LLM's translation and multilingual reasoning, and it finds that keeping parallel sentences adjacent and putting them at the end helps most. The qualitative story likely holds; the precise ranking is weaker than the paper claims.\n\nWhat's new: seven settings (no parallel, monolingual, non-adjacent, first, distributed, last-all, last-uni) matched for data size and order. The paper shows paired parallel data beats an equal amount of foreign monolingual data, that non-adjacent placement helps less than adjacent, and that unidirectional parallel training yields models that can't translate the reverse direction—worse than no parallel at all. That last finding is striking and well demonstrated. They also release code, checkpoints, and data, which is a real asset. The common-sense reasoning evaluation is a plus: parallel data helps non-English reasoning, not just translation.\n\nWhere it gets soft. First, and most important: the checkpoint selection. Checkpoints are saved every 5k steps, but for the first 5k of PARALLELFIRST and the last 5k of PARALLELLAST they save every 1k steps, then pick the best dev BLEU. That gives those two settings five times more checkpoint candidates than the others. For PARALLELLAST, the recommended setting, that asymmetry can inflate scores. The margins in Table 3 between PARALLELLAST and PARALLELDISTRIBUTED are large in some directions but small or reversed in others; without equal checkpoint density we don't know how the ranking would hold. The stress-test note is right about this. Second, footnote 4 admits PARALLELLAST settings train on 166B/164B tokens instead of 167B, so \"consistent training set size and order\" is not literally true. Minor, but worth flagging. Third, single runs mean the paired randomization significance test compares two models, not training noise. That's a known limitation, but it makes \"significant\" less secure.\n\nThe Section 6.4 word-frequency analysis is more of a correlation; it builds a dictionary from the same test sets it evaluates on, so treat it as illustrative, not causal.\n\nOverall: the central qualitative finding—that parallel data matters and paired form beats scattered monolingual text—holds up well across benchmarks. The \"end-of-training best\" ranking is plausible but not fully established. This paper deserves peer review; a serious reviewer should ask for seed variance or at least a re-analysis equalizing checkpoint density. I'd cite it for the controlled comparison and the unidirectional finding, and bring it to reading group. Engage with it, but treat the ranking with caution until the checkpoint issue is addressed.","headline":"A useful controlled study of parallel-data placement for multilingual LLMs, but checkpoint-selection asymmetry and single runs weaken the 'end-of-training best' ranking; deserves review with revisions.","tokens_in":17966,"tokens_out":2414,"would_cite":true,"duration_ms":22430,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Parallel sentences added at the end of training markedly improve an LLM's translation and non-English reasoning.","keywords":["parallel data","multilingual LLMs","machine translation","second-stage training","catastrophic forgetting","cross-lingual transfer","reversal curse","common-sense reasoning"],"falsifier":"Train the same seven data-placement settings with several random seeds at the same 167B-token budget and compare the selected checkpoints on held-out translation and reasoning. The central claims would be contradicted if the parallel-last strategy does not consistently beat parallel-distributed, or if a model trained only on English-to-Chinese translates Chinese-to-English well above the no-parallel baseline.","tokens_in":16937,"feed_emoji":"🌐","tokens_out":8720,"duration_ms":74870,"temperature":0.7,"pith_summary":"This paper asks whether paired parallel sentences still matter for decoder-only language models, whose translation skill is often attributed to scale alone. By holding the model, total tokens, and data order fixed across seven training strategies, it claims that parallel data improve translation and non-English common-sense reasoning more than the same amount of foreign monolingual text. The placement of the pairs is decisive: adding them at the end of training is the most effective strategy, while adding them at the beginning wastes the benefit through catastrophic forgetting. The paper also claims that a model trained on only one translation direction cannot translate the reverse direction, scoring worse than a model that never saw parallel data.","feed_headline":"Parallel data late in training boosts LLM multilingual skills","feed_subtitle":"A controlled 1.1B-parameter study shows paired sentences beat monolingual text when added at the end of pretraining.","key_machinery":"The load-bearing mechanism is data placement rather than data quantity. The same roughly 4.5 billion tokens of parallel sentences, formatted as adjacent source-target chunks with language labels, are inserted into a fixed 167-billion-token training stream in different positions: at the beginning, distributed throughout, shuffled into non-adjacent pairs, or at the end as a second-stage training block. This design isolates the bilingual-signal effect, namely whether the model ever sees a sentence next to its translation, from the effect of merely adding foreign text. Checkpoint selection by development-set BLEU and paired approximate randomization then turn placement differences into a ranking of training strategies.","core_discovery":"The central claim, as the paper states it, is that decoder-only LLMs gain substantially more multilingual ability from paired parallel sentences than from equivalent monolingual text, and that the gains depend on where the pairs appear in the training stream. In the controlled comparison, the model trained with adjacent English-Chinese and English-Indonesian sentence pairs as a final training stage reached BLEU scores of roughly 36 on English-to-Indonesian and Indonesian-to-English, compared with under 3 for the no-parallel baseline, and it produced the best Indonesian and Chinese common-sense reasoning scores. Training with the same parallel pairs distributed throughout the corpus beat both monolingual-only training and training with the pairs shuffled into non-adjacent sentences, which isolates sentence-to-sentence alignment as the active ingredient. Putting the pairs at the beginning produced only temporary translation gains that disappeared once later monolingual data washed them out. Finally, a model trained on just one direction (for example, English-to-Chinese) became a stronger one-way translator but was essentially unable to translate the opposite direction, with near-zero BLEU scores that are worse than the no-parallel model.","pith_inferences":["A natural extension is to sweep the position of the parallel block continuously through training; the paper's data predict that later placement should monotonically improve translation, which is testable on the released checkpoints.","The one-direction failure suggests that emergent reverse translation should not be assumed at larger scales either; a cheap instruction-tuned reverse-direction stage is a direct add-on this result says is necessary for bidirectional use.","Since each setting was trained once, a multi-seed replication at the same 1.1B scale would size the run-to-run noise and check whether the parallel-last versus parallel-distributed gap survives.","The bilingual-signal frequency result implies a controllable volume effect: doubling the parallel tokens at the end of training should lift low-frequency words across the translation-accuracy threshold, which can be measured without changing the architecture."],"forward_implications":["Multilingual decoder LLMs should keep parallel sentence pairs intact and schedule them as a final training stage rather than scattering them or discarding the English half.","Unidirectional parallel training is a recipe for a specialized machine-translation model, not a general multilingual model, because the reverse direction and other language pairs remain untranslatable.","Parallel data improve non-English common-sense reasoning, so they should be counted as general multilingual capability rather than translation-only data.","The amount of incidental bilingual context in pretraining data matters: words with such context are translated correctly more often, which argues for deliberately enriching pretraining data with parallel-like contexts.","At this scale, noisy parallel pairs are tolerable: quality filtering helped only some zero-shot directions and slightly hurt few-shot scores, so aggressive filtering is not clearly required."],"supporting_citations":[{"why":"Supplies the thesis that incidental bilingual signals, especially translation pairs, drive LLM translation capability, motivating the parallel-data intervention.","marker":"Briakou et al. (2023)"},{"why":"Provides the second-phase pretraining result that supports the paper's parallel-last strategy.","marker":"Gururangan et al. (2020)"},{"why":"Supplies the reversal-curse explanation for the unidirectional model's failure to translate the reverse direction.","marker":"Berglund et al. (2024)"},{"why":"Defines BLEU, the metric used for checkpoint selection and translation evaluation.","marker":"Papineni et al. (2002)"},{"why":"Provides SacreBLEU, the evaluation tool used to report translation scores.","marker":"Post (2018)"},{"why":"Supplies the prompt style and evaluation setup used for zero-shot and few-shot translation tests.","marker":"Xu et al. (2024)"},{"why":"Provides the Flores-200 development sets and the framing that parallel data are increasingly available for lower-resource languages.","marker":"Costa-jussà et al. (2022)"},{"why":"Provides the WMT-2022 Chinese-English parallel training data used in the experiments.","marker":"Kocmi et al. (2022)"},{"why":"Provides the WMT-2021 Indonesian-English parallel training data used in the experiments.","marker":"Wenzek et al. (2021)"},{"why":"Supplies the concept of catastrophic forgetting used to explain why parallel-first training loses its gains.","marker":"McCloskey and Cohen (1989)"}],"fun_headline_variants":["Just go parallel: late paired data lifts LLM multilingual skills","Parallel data late in training beats monolingual for LLM","Adjacent parallel pairs late in training boost LLM languages","One-way parallel training yields one-way translators in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of the seven strategies assumes that one training run per setting is enough, so the reported differences are not artifacts of run-to-run training noise.","fun_headline_variants_meta":{"raw":{"variants":["Just go parallel: late paired data lifts LLM multilingual skills","Parallel data late in training beats monolingual for LLM","Adjacent parallel pairs late in training boost LLM languages","One-way parallel training yields one-way translators in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0008,"raw_usage":{"total_tokens":3503,"prompt_tokens":917,"completion_tokens":2586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2517}},"tokens_in":533,"tokens_out":2586,"duration_ms":17745,"temperature":1.0,"reasoning_tokens":2517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:06:02.156029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same seven data-placement settings with several random seeds at the same 167B-token budget and compare the selected checkpoints on held-out translation and reasoning. The central claims would be contradicted if the parallel-last strategy does not consistently beat parallel-distributed, or if a model trained only on English-to-Chinese translates Chinese-to-English well above the no-parallel baseline.","supporting_citations":[{"cited_title":"A is B","cited_arxiv_id":null,"evidence_quote":"Supplies the reversal-curse explanation for the unidirectional model's failure to translate the reverse direction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides SacreBLEU, the evaluation tool used to report translation scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WMT-2022 Chinese-English parallel training data used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WMT-2021 Indonesian-English parallel training data used in the experiments."}],"review_version":1}