Pith. sign in

REVIEW 4 major objections 5 minor 76 references

Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Parallel sentences added at the end of training markedly improve an LLM's translation and non-English reasoning.

desk verdict A useful controlled study of parallel-data placement for multilingual LLMs, but checkpoint-selection asymmetry and single runs weaken the 'end-of-training best' ranking; deserves review with revisions. read the letter →

arxiv 2506.13044 v1 pith:K4AA6FHM submitted 2025-06-16 cs.CL cs.AI

Muhammad Reza Qorib , Junyi Li , Hwee Tou Ng This is my paper · ORCID
classification cs.CLcs.AI
keywords paralleldatamultilingualLLMsmachinetranslationsecond-stagetrainingcatastrophicforgettingcross-lingualtransferreversalcursecommon-sensereasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether paired parallel sentences still matter for decoder-only language models, whose translation skill is often attributed to scale alone. By holding the model, total tokens, and data order fixed across seven training strategies, it claims that parallel data improve translation and non-English common-sense reasoning more than the same amount of foreign monolingual text. The placement of the pairs is decisive: adding them at the end of training is the most effective strategy, while adding them at the beginning wastes the benefit through catastrophic forgetting. The paper also claims that a model trained on only one translation direction cannot translate the reverse direction, scoring worse than a model that never saw parallel data.

What carries the argument

The load-bearing mechanism is data placement rather than data quantity. The same roughly 4.5 billion tokens of parallel sentences, formatted as adjacent source-target chunks with language labels, are inserted into a fixed 167-billion-token training stream in different positions: at the beginning, distributed throughout, shuffled into non-adjacent pairs, or at the end as a second-stage training block. This design isolates the bilingual-signal effect, namely whether the model ever sees a sentence next to its translation, from the effect of merely adding foreign text. Checkpoint selection by development-set BLEU and paired approximate randomization then turn placement differences into a ranking of training strategies.

What would settle it

Train the same seven data-placement settings with several random seeds at the same 167B-token budget and compare the selected checkpoints on held-out translation and reasoning. The central claims would be contradicted if the parallel-last strategy does not consistently beat parallel-distributed, or if a model trained only on English-to-Chinese translates Chinese-to-English well above the no-parallel baseline.

Watch

Extended reading notes

Core claim

The central claim, as the paper states it, is that decoder-only LLMs gain substantially more multilingual ability from paired parallel sentences than from equivalent monolingual text, and that the gains depend on where the pairs appear in the training stream. In the controlled comparison, the model trained with adjacent English-Chinese and English-Indonesian sentence pairs as a final training stage reached BLEU scores of roughly 36 on English-to-Indonesian and Indonesian-to-English, compared with under 3 for the no-parallel baseline, and it produced the best Indonesian and Chinese common-sense reasoning scores. Training with the same parallel pairs distributed throughout the corpus beat both monolingual-only training and training with the pairs shuffled into non-adjacent sentences, which isolates sentence-to-sentence alignment as the active ingredient. Putting the pairs at the beginning produced only temporary translation gains that disappeared once later monolingual data washed them out. Finally, a model trained on just one direction (for example, English-to-Chinese) became a stronger one-way translator but was essentially unable to translate the opposite direction, with near-zero BLEU scores that are worse than the no-parallel model.

Load-bearing premise

The ranking of the seven strategies assumes that one training run per setting is enough, so the reported differences are not artifacts of run-to-run training noise.

Editorial extensions

If this is right

  • Multilingual decoder LLMs should keep parallel sentence pairs intact and schedule them as a final training stage rather than scattering them or discarding the English half.
  • Unidirectional parallel training is a recipe for a specialized machine-translation model, not a general multilingual model, because the reverse direction and other language pairs remain untranslatable.
  • Parallel data improve non-English common-sense reasoning, so they should be counted as general multilingual capability rather than translation-only data.
  • The amount of incidental bilingual context in pretraining data matters: words with such context are translated correctly more often, which argues for deliberately enriching pretraining data with parallel-like contexts.
  • At this scale, noisy parallel pairs are tolerable: quality filtering helped only some zero-shot directions and slightly hurt few-shot scores, so aggressive filtering is not clearly required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to sweep the position of the parallel block continuously through training; the paper's data predict that later placement should monotonically improve translation, which is testable on the released checkpoints.
  • The one-direction failure suggests that emergent reverse translation should not be assumed at larger scales either; a cheap instruction-tuned reverse-direction stage is a direct add-on this result says is necessary for bidirectional use.
  • Since each setting was trained once, a multi-seed replication at the same 1.1B scale would size the run-to-run noise and check whether the parallel-last versus parallel-distributed gap survives.
  • The bilingual-signal frequency result implies a controllable volume effect: doubling the parallel tokens at the end of training should lift low-frequency words across the translation-accuracy threshold, which can be measured without changing the architecture.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a controlled study of how parallel sentence pairs affect the multilingual capabilities of decoder-only LLMs. Using a 1.1B TinyLlama model trained on about 167B tokens, it compares seven data-placement strategies: no parallel data, multilingual monolingual data, non-adjacent parallel data, parallel data at the start, parallel data distributed throughout, parallel data at the end in all directions, and parallel data at the end in one direction. Translation is evaluated on WMT-23 and Flores-200 devtest, and common-sense reasoning on English, Chinese, and Indonesian benchmarks. The paper claims that parallel data improves translation and non-English reasoning, that placing parallel data at the end is the most effective placement, that unidirectional parallel training does not transfer to the reverse direction, and that the amount of incidental bilingual signal correlates with translation quality.

Significance. If the ranking of data-placement strategies holds, the paper gives concrete and actionable guidance for multilingual LLM training: preserve the parallel format, add parallel data at the end, and consider unidirectional specialization for translation systems. The release of code, checkpoints, and data, together with evaluation on external test sets, is a genuine strength. The reverse-direction failure of unidirectional models is a striking and falsifiable finding. However, the central placement claim is weakened by an asymmetric checkpoint-selection procedure and by single-run training without seed variation, so the significance is contingent on additional controlled evidence.

major comments (4)
  1. [Section 4, checkpoint selection] The checkpoint-selection procedure systematically favors PARALLELLAST and PARALLELFIRST. The paper states that checkpoints are saved every 5,000 steps, but for the first 5,000 steps of PARALLELFIRST and the last 5,000 steps of PARALLELLAST settings, checkpoints are saved every 1,000 steps, and then the checkpoint with the highest average dev BLEU is selected for every setting. Thus PARALLELLAST and PARALLELFIRST are best-of-five selections over a 5k-step window, while the other settings are best-of-one at 5k boundaries. This asymmetry inflates the apparent advantage of the recommended PARALLELLAST strategy. In Table 3, the zero-shot margins between PARALLELLAST(ALL) and PARALLELDISTRIBUTED range from about 3 to 14 BLEU points (e.g., 35.91 vs. 21.95 for EN→ID and 10.73 vs. 7.40 for ZH→EN), so the conclusion that 'training at the end is the most effective approach' is not supported as a controlled comparison. The authors should evaluate all settings with the same checkpoint density, or compare final checkpoints rather than best-of-variable checkpoints.
  2. [Section 3 vs. footnote 4] The token-count control is not exact. Section 3 says 'we maintain the order and quantity of training data across all experiments' and 'preserve a consistent training set size and order,' but footnote 4 reports that PARALLELLAST(ALL) is trained on 166B tokens and PARALLELLAST(UNI) on 164B tokens instead of 167B, because not all parallel data are consumed. This means the PARALLELLAST treatments differ from the NO PARALLEL baseline not only in data placement but also in total token count. The qualitative benefit of parallel data may survive this confound, but the precise ranking in Table 3 and the claim of a consistent control do not.
  3. [Section 4 and Section 5, statistical significance] Each experimental setting is trained only once, so the paired approximate randomization test in Section 5 does not bound training stochasticity. That test compares two single models on test sentences; it cannot tell whether the difference between, say, PARALLELDISTRIBUTED and PARALLELLAST(ALL) would persist under a different random seed or data-order shuffle. Larger, non-overlapping runs are needed for the central ranking, or the paper should present its conclusions as single-run observations and soften the comparative claims accordingly.
  4. [Section 6.4, word-level bilingual-signal analysis] The word-level analysis is circular. The paper builds a word translation dictionary by aligning source and target sentences from the WMT-2022 test set (Chinese-English) and the Flores-200 dev set (Indonesian-English), then uses that same dictionary and those same references to judge whether the model's output contains the correct translation. A word's 'correct translation ratio' is therefore partly determined by the dictionary construction procedure on the evaluation sentences themselves. To support the claim that incidental bilingual signal frequency affects translation capability, the dictionary should be induced from a separate held-out parallel corpus and then applied to a disjoint test set.
minor comments (5)
  1. [Section 6.2, threshold selection] The CometKiwi thresholds (0.42 for Chinese-English and 0.58 for Indonesian-English) are described as set 'based on manual observation'; the paper should state how many examples were inspected and whether the thresholds were fixed before or after seeing the translation results, since threshold choice can affect the filtered-data comparison.
  2. [Table 3 and Table 5, presentation of PARALLELLAST(UNI)] In Table 3, PARALLELLAST(UNI) is shown as one row, but it represents four separately trained unidirectional models; the table should make explicit that each cell in that row comes from a different model.
  3. [Figure 2] Figure 2 should state whether the BLEU curve is computed on the development set or the test set, and should clearly identify which curve corresponds to NOPARALLEL and which to PARALLELFIRST.
  4. [Section 4, development set terminology] Using the WMT-2022 test set and Flores-200 dev set as the checkpoint-selection development set is unusual and could confuse readers; consider renaming them 'selection sets' or clearly stating that no held-out test data are used for selection.
  5. [Section 5, English common-sense reasoning] The decline in English common-sense accuracy for PARALLELLAST(ALL) (50.44 for NO PARALLEL to 49.08 for PARALLELLAST(ALL) in Table 4) is reported descriptively; a discussion of whether this is within expected single-run variation would be useful given that only one run is available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: conclusions are controlled empirical comparisons on external benchmarks, not derivations that reduce to their inputs.

full rationale

The paper is an empirical comparison study with no fitted derivation chain: its conclusions are supported by controlled training runs (NOPARALLEL, MULTILINGUAL, PARALLEL*, etc.) evaluated on external test sets (WMT-23, Flores-200) and standard reasoning benchmarks. The claim that parallel data helps, and that placement at the end is most effective, is a reported comparison of independent runs rather than a quantity defined by its inputs; footnote 4's acknowledged token-count differences and the single-run design are experimental limitations, not circular reductions. The Section 6.4 word-frequency analysis tags words as having bilingual signals when they co-occur with their gold translations in a 2048-token window and then checks whether those translations appear in model output; this is a diagnostic correlation that can fail in both directions and does not define the outcome it explains. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation; the PaLM-style parallel-data format is an input design choice, not a result claimed to be derived. The checkpoint-selection asymmetry (denser saving for PARALLELLAST/PARALLELFIRST) is a potential validity concern for the ranking of strategies, but it does not make any reported number equal to its own input by construction. Therefore no circular step is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No invented entities or new theoretical constructs are introduced. The free parameters are limited to the manually chosen CometKiwi thresholds in a secondary filtering experiment. The axioms are standard domain assumptions about evaluation tools (fastText, SimAlign, CometKiwi) and about the representativeness of a single model and data mixture. The central claim does not introduce new fitted quantities beyond trained model weights.

free parameters (2)
  • CometKiwi filtering threshold for Chinese-English pairs = 0.42
    Set by manual observation in Section 6.2 to filter low-quality parallel sentences. This is a hand-chosen threshold in a secondary experiment, not used in the main placement comparison.
  • CometKiwi filtering threshold for Indonesian-English pairs = 0.58
    Also chosen manually in Section 6.2; reduces the Indonesian-English parallel corpus from 54.1M to 15.6M sentences. The choice could affect the filtering conclusions.
assumptions (5)
  • domain assumption TinyLlama continued pretraining on SlimPajama is representative of how multilingual LLMs are built.
    The experiments use one base model and one English-heavy corpus; the authors assume results generalize to other decoder-only LLMs and data mixtures (Section 4).
  • domain assumption fastText per-word language detection accurately measures training data language composition.
    Appendix A.1 uses fastText to conclude incidental bilingual data is minimal; errors in language detection could change the interpretation of the NOPARALLEL baseline.
  • domain assumption SimAlign word alignments are accurate enough to build the bilingual-signal dictionary.
    Section 6.4 relies on SimAlign to identify word translations in test and dev sets; alignment errors would propagate into the word-level frequency analysis.
  • domain assumption CometKiwi-2022 scores are a valid measure of parallel-data quality for filtering.
    Section 6.2 uses CometKiwi as the quality signal; if these scores do not track human-perceived translation quality, the filtering results are not interpretable.
  • domain assumption The reversal curse is the correct explanation for unidirectional models' failure on the opposite direction.
    Section 6.1 attributes the one-direction-only failure to the reversal curse of Berglund et al. (2024); this is an interpretive leap beyond the paper's own measurements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models." pith.science (2026). https://pith.science/paper/K4AA6FHM

@misc{pith2026250613044,
  author       = {Pith},
  title        = {Pith review of: Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4AA6FHM}},
  note         = {Machine review of arXiv:2506.13044}
}
read the original abstract

Large language models (LLMs) have demonstrated impressive translation capabilities even without being explicitly trained on parallel data. This remarkable property has led some to believe that parallel data is no longer necessary for building multilingual language models. While some attribute this to the emergent abilities of LLMs due to scale, recent work suggests that it is actually caused by incidental bilingual signals present in the training data. Various methods have been proposed to maximize the utility of parallel data to enhance the multilingual capabilities of multilingual encoder-based and encoder-decoder language models. However, some decoder-based LLMs opt to ignore parallel data instead. In this work, we conduct a systematic study on the impact of adding parallel data on LLMs' multilingual capabilities, focusing specifically on translation and multilingual common-sense reasoning. Through controlled experiments, we demonstrate that parallel data can significantly improve LLMs' multilingual capabilities.

Figures

Figures reproduced from arXiv: 2506.13044 by the authors.

Figure 1
Figure 1. Macro-average of the common-sense reason [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Chinese-to-English translation performance [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Indonesian-to-English translation perfor [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 47 canonical work pages

  1. [1]

    Mohammed Al - Maamari, Mehdi Ben Amor, and Michael Granitzer. 2024. Mixture of modular experts: Distilling knowledge from a multilingual teacher into specialized modular language models. arXiv preprint arXiv:2407.19610

    arXiv 2024

  2. [2]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \' e rouane Debbah, \' E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The Falcon series of open language models. arXiv preprint arXiv:2311.16867

    arXiv 2023

  3. [3]

    Duarte Miguel Alves, Jos \'e Pombal, Nuno M Guerreiro, Pedro Henrique Martins, Jo \ a o Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, Jos \'e G. C. de Souza, and Andre Martins. 2024. https://openreview.net/forum?id=EHPns3hVkj Tower: An open multilingual large language model for translation-related tasks ....

    work page 2024

  4. [4]

    A is B

    Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2024. https://openreview.net/forum?id=GPKTIktA0k The reversal curse: LLMs trained on “ A is B ” fail to learn “ B is A ” . In Proceedings of ICLR

    work page 2024

  5. [5]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of ICML, pages 2397--2430

    work page 2023

  6. [6]

    Steven Bird and Edward Loper. 2004. https://aclanthology.org/P04-3031/ NLTK : The natural language toolkit . In Proceedings of the ACL Interactive Poster and Demonstration Sessions , pages 214--217

    work page 2004

  7. [7]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA : Reasoning about physical commonsense in natural language. In Proceedings of AAAI, pages 7432--7439

    work page 2020

  8. [8]

    Eleftheria Briakou, Colin Cherry, and George Foster. 2023. https://doi.org/10.18653/v1/2023.acl-long.524 Searching for needles in a haystack: On the role of incidental bilingualism in P a LM `s translation capability . In Proceedings of ACL, pages 9432--9452

    doi:10.18653/v1/2023.acl-long.524 2023

Show all 76 references
  1. [9]

    Steven Cao, Nikita Kitaev, and Dan Klein. 2020. https://openreview.net/forum?id=r1xCMyBtPS Multilingual alignment of contextual word representations . In Proceedings of ICLR

    2020

  2. [10]

    Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xinnian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. 2025. xCoT : Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. In Proceedings of AAAI

    2025

  3. [11]

    Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, and Furu Wei. 2021 a . https://doi.org/10.18653/v1/2021.emnlp-main.125 m T 6: Multilingual pretrained text-to-text transformer with translation pairs . In Proceedings of EMNLP...

    2021 doi

  4. [12]

    Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian - Ling Mao, Heyan Huang, and Furu Wei. 2021 b . Improving pretrained cross-lingual language models via self-labeled word alignment. In Proceedings of ACL, pages 3418--3430

    2021

  5. [13]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and et al. 2023. PaLM : Scaling language modeling with pathways. JMLR, 24(1)

    2023

  6. [14]

    Christopher Clark, Kenton Lee, Ming - Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ : Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL, pages 2924--2936

    2019

  7. [15]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC , the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  8. [16]

    Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Proceedings of NeurIPS

    2019

  9. [17]

    Bowman, Holger Schwenk, and Veselin Stoyanov

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI : Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, pages 2475--2485

    2018

  10. [18]

    Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672

    2022 arXiv

  11. [19]

    Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2024. Do multilingual language models think better in English? In Proceedings of NAACL, pages 550--564

    2024

  12. [20]

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tusha...

    2024

  13. [21]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.acl-main.740 Don ' t stop pretraining: Adapt language models to domains and tasks . In Proceedings of ACL, pages 8342--8360

    2020 doi

  14. [22]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs : Improving multilingual capability by cross-lingual-thought prompting. In Findings of EMNLP, pages 12365--12394

    2023

  15. [23]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H \'e rve J \'e gou, and Tomas Mikolov. 2016. FastText.zip : Compressing text classification models. arXiv preprint arXiv:1612.03651

    2016 arXiv

  16. [24]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of EACL, pages 427--431

    2017

  17. [25]

    Tom Kocmi, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Nov \'a k, Ma...

    2022

  18. [26]

    Philipp Koehn. 2005. https://aclanthology.org/2005.mtsummit-papers.11/ E uroparl: A parallel corpus for statistical machine translation . In Proceedings of Machine Translation Summit X: Papers, pages 79--86

    2005

  19. [27]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022 doi

  20. [28]

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. https://doi.org/10.1162/tacl_a_00343 Multilingual denoising pre-training for neural machine translation . TACL, 8:726--742

    2020 doi

  21. [29]

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. 2024. https://openreview.net/forum?id=QdWhj0QZFw LLM360 : Towards fully transparent open-source LLMs . In Proceedings of COLM

    2024

  22. [30]

    Rutherford, and Sarana Nutanong

    Lalita Lowphansirikul, Charin Polpanumas, Attapol T. Rutherford, and Sarana Nutanong. 2020. https://api.semanticscholar.org/CorpusID:220381366 A large E nglish– T hai parallel corpus from the web and machine-generated text . Language Resources and Evaluation, 56:477 -- 499

    2020

  23. [31]

    Jinliang Lu, Yu Lu, and Jiajun Zhang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.190 Take a closer look at multilinguality! I mprove multilingual pre-training using monolingual corpora only . In Findings of EMNLP, pages 2891--2907

    2023 doi

  24. [32]

    Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. 2021. https://doi.org/10.18653/v1/2021.acl-long.308 VECO : Variable and flexible cross-lingual pre-training for language understanding and generation . In Proceedings of ACL, pages 3980--3994

    2021 doi

  25. [33]

    Zhuoyuan Mao and Yen Yu. 2024. https://doi.org/10.18653/v1/2024.loresmt-1.1 Tuning LLM s with contrastive alignment instructions for machine translation in unseen, low-resource languages . In Proceedings of LoResMT, pages 1--25

    2024 doi

  26. [34]

    Michael McCloskey and Neal J. Cohen. 1989. https://doi.org/10.1016/S0079-7421(08)60536-8 Catastrophic interference in connectionist networks: The sequential learning problem . Psychology of Learning and Motivation, 24:109--165

    1989 doi

  27. [35]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of EMNLP, pages 2381--2391

    2018

  28. [36]

    Roberto Navigli, Simone Conia, and Bj\" o rn Ross. 2023. https://doi.org/10.1145/3597307 Biases in large language models: Origins, inventory, and discussion . J. Data and Information Quality, 15(2):1--21

    2023 doi

  29. [37]

    Sina Bagheri Nezhad and Ameeta Agrawal. 2024. https://doi.org/10.18653/v1/2024.vardial-1.2 What drives performance in multilingual language models? In Proceedings of VarDial, pages 16--27

    2024 doi

  30. [38]

    Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias St \" u rmer, and Ilias Chalkidis. 2023. LEXTREME: A multi-lingual and multi-task benchmark for the legal domain. In Findings of EMNLP, pages 3016--3054

    2023

  31. [39]

    Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.3 ERNIE - M : Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora . In Proceedings of EMNLP, pages 27--38

    2021 doi

  32. [40]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 BLEU : A method for automatic evaluation of machine translation . In Proceedings of ACL, pages 311--318

    2002

  33. [41]

    Edoardo Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of EMNLP, pages 2362--2376

    2020

  34. [42]

    Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of WMT, pages 186--191

    2018

  35. [43]

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of EMNLP, pages 2695--2709

    2023

  36. [44]

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925

    2024 arXiv

  37. [45]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://openai.com/index/better-language-models/ Language models are unsupervised multitask learners

    2019

  38. [46]

    Leonardo Ranaldi, Giulia Pucci, Barry Haddow, and Alexandra Birch. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.678 Empowering multi-step reasoning across languages via program-aided language models . In Proceedings of EMNLP, pages 12171--12187

    2024 doi

  39. [47]

    Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G

    Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.60/ C omet K iwi: IST - U ...

    2022

  40. [48]

    Masoud Jalili Sabet, Philipp Dufter, Fran c ois Yvon, and Hinrich Sch \" u tze. 2020. SimAlign : High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of EMNLP, pages 1627--1643

    2020

  41. [49]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande : An adversarial Winograd schema challenge at scale. In Proceedings of AAAI, pages 8732--8740

    2020

  42. [50]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagn \' e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \' e , Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wa...

    2022 arXiv

  43. [51]

    Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of NAACL, pages 1599--1613

    2019

  44. [52]

    Kirill Semenov, Vil \'e m Zouhar, Tom Kocmi, Dongdong Zhang, Wangchunshu Zhou, and Yuchen Eleanor Jiang. 2023. https://doi.org/10.18653/v1/2023.wmt-1.54 Findings of the WMT 2023 shared task on machine translation with terminologies . In Proceedings of WMT, pages 663--671

    2023 doi

  45. [53]

    Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andr...

    2023 arXiv

  46. [54]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama : A 627B token cleaned and deduplicated version of RedPajama

    2023

  47. [55]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji - Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of ACL, pages 5701--5715

    2024

  48. [56]

    Alexey Tikhonov and Max Ryabinin. 2021. https://doi.org/10.18653/v1/2021.findings-acl.310 I t`s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning . In Findings of ACL, pages 3534--3546

    2021 doi

  49. [57]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 arXiv

  50. [58]

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2024. OpenChat : A dvancing open-source language models with mixed-quality data. In Proceedings of ICLR

    2024

  51. [59]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of W-NUT, pages 94--106

    2017

  52. [60]

    Wen - Yi and David Mimno

    Andrea W. Wen - Yi and David Mimno. 2023. Hyperpolyglot LLMs : Cross-lingual interpretability in token embeddings. In Proceedings of EMNLP, pages 1124--1131

    2023

  53. [61]

    Guillaume Wenzek, Vishrav Chaudhary, Angela Fan, Sahir Gomez, Naman Goyal, Somya Jain, Douwe Kiela, Tristan Thrush, and Francisco Guzm \'a n. 2021. https://aclanthology.org/2021.wmt-1.2/ Findings of the WMT 2021 shared task on large-scale multilingual machine translation . In ...

    2021

  54. [62]

    Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. https://openreview.net/forum?id=farT6XXntP A paradigm shift in machine translation: Boosting translation performance of large language models . In Proceedings of ICLR

    2024

  55. [63]

    Haoran Xu, Kenton Murray, Philipp Koehn, Hieu Hoang, Akiko Eriguchi, and Huda Khayrallah. 2025. https://openreview.net/forum?id=csbf1p8xUq X- ALMA : Plug & play modules and adaptive rejection for quality translation at scale . In Proceedings of ICLR

    2025

  56. [64]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al - Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5 : A massively multilingual pre-trained text-to-text transformer. In Proceedings of NAACL, pages 483--498

    2021

  57. [65]

    Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. https://arxiv.org/abs/2305.18098 Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages . arXiv preprint arXiv:2305.18098

    2023 arXiv

  58. [66]

    Xiang Yue, Yueqi Song, Akari Asai, Simran Khanuja, Anjali Kantharuban, Seungone Kim, Jean de Dieu Nyandwi, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2025. https://openreview.net/forum?id=a3g2l4yEys Pangea: A fully open multilingual multimodal LLM for 39...

    2025

  59. [67]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag : Can a machine really finish your sentence? In Proceedings of ACL, pages 4791--4800

    2019

  60. [68]

    Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. 2024. https://doi.org/10.1609/aaai.v38i17.29920 Teaching large language models to translate with comparison . In Proceedings of AAAI, pages 19488--19496

    2024 doi

  61. [69]

    Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. https://doi.org/10.18653/v1/2020.acl-main.148 Improving massively multilingual neural machine translation and zero-shot translation . In Proceedings of ACL, pages 1628--1639

    2020 doi

  62. [70]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024 a . TinyLlama : An open-source small language model. arXiv preprint arXiv:2401.02385

    2024 arXiv

  63. [71]

    Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024 b . Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages. In Proceedings of ACL, pages 11189--11204

    2024

  64. [72]

    Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond E nglish: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055

    2024 arXiv

  65. [73]

    Shaolin Zhu, Supryadi, Shaoyang Xu, Haoran Sun, Leiyu Pan, Menglong Cui, Jiangcun Du, Renren Jin, António Branco, and Deyi Xiong. 2024 a . Multilingual large language models: A systematic survey. arXiv preprint arXiv:2411.11072

    2024 arXiv

  66. [74]

    Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024 b . Question translation training for better multilingual reasoning. In Findings of ACL, pages 8411--8423

    2024

  67. [75]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  68. [76]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.