Pith. sign in

REVIEW 3 major objections 4 minor 59 references

Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SALT recycles target-language PLM embeddings to outperform existing cross-lingual transfer methods on German, Arabic, and Vietnamese.

desk verdict SALT is a clean, novel initialization trick with mostly positive results, but the per-token regression is likely rank-deficient and the 'significantly outperforms' claim needs more statistical and ablative support. read the letter →

arxiv 2505.10945 v2 pith:N6ZPYCYP submitted 2025-05-16 cs.CL cs.AI

Seungyoon Lee , Seongtae Hong , Hyeonseok Moon , Heuiseok Lim This is my paper · ORCID
classification cs.CLcs.AI
keywords SALTcross-lingualtransferembeddinginitializationpre-trainedlanguagemodelslinearleastsquaresvocabularyreplacementadaptationmultilingualLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes SALT, a way to move an English-centric large language model (LLM) into a target language by recycling the embeddings of smaller, target-language pretrained models such as BERT, GPT, or T5. For every token the two vocabularies do not share, SALT finds the target token's nearest neighbors among shared tokens using fastText similarities, then fits a per-token least-squares linear map from the pretrained language model's embedding space into the LLM's embedding space. The paper argues that this preserves the target language's semantic richness while placing new embeddings where the LLM expects them, so the transferred model adapts faster, reaches lower language-modeling loss, and scores higher on downstream tasks than the FOCUS, OFA, and multivariate baselines. The claim matters because it offers a parameter-light, initialization-only route to convert large English-centric models into language-specific ones without training embeddings from scratch.

What carries the argument

The central object is a per-token linear least-squares transform. For each non-shared target token $v_{t_i}$, SALT uses fastText cosine similarity to rank shared vocabulary tokens, applies Sparsemax to select the dynamic top-$k$ nearest neighbors, and stacks their PLM embeddings $E'_{t_i}$ and source-LLM embeddings $E'_{s_i}$. It then solves $\arg\min_{X\in\mathbb{R}^{h_t\times h_s}}\|E'_{t_i}X_{t_i}-E'_{s_i}\|$, whose closed-form solution is $X_{t_i}=E'^{+}_{t_i}E'_{s_i}$ via the pseudo-inverse. This unique map per token projects that token's PLM embedding into the LLM's embedding space, preserving target-language semantic structure while aligning with the source model's learned representation space.

What would settle it

Take a language pair where the source and target vocabularies share very few tokens, fit SALT on those anchors, and compare the resulting model's downstream accuracy and final training loss to multivariate initialization under identical training; if SALT does not improve, the per-token regression did not generalize beyond its anchors.

Watch

Extended reading notes

Core claim

The paper's central claim is that recycling target-language pretrained-language-model (PLM) embeddings through per-token linear regression gives a better initialization for cross-lingual transfer than reweighting or resampling the source LLM's own embeddings. In the reported experiments, SALT achieves the highest average accuracy on knowledge benchmarks (ARC, HellaSwag, TruthfulQA) and reading-comprehension tasks (MLQA, XQuAD, Belebele) across German, Arabic, and Vietnamese for both Gemma-2b and XGLM-1.7b source models. Models initialized with SALT also converge faster and end with lower causal language modeling loss during language-adaptive continual pretraining, and they retain English question-answering ability better than the baselines. The paper further shows that the PLM supplying the embeddings does not have to be a decoder, since BERT and T5 target models also perform competitively, indicating that older small models remain useful embedding sources for contemporary LLMs.

Load-bearing premise

The method assumes that a linear map fitted on shared tokens places unseen target-language tokens correctly in the LLM's embedding space; if the PLM and LLM spaces are not linearly related for non-shared tokens, SALT loses its advantage.

Editorial extensions

If this is right

  • SALT-initialized models reach the lowest final training loss and converge faster than FOCUS, OFA, and multivariate initialization in every language tested.
  • SALT improves target-language performance on knowledge-based benchmarks and machine reading comprehension, with the largest gains on HellaSwag and generation-based MRC tasks.
  • SALT preserves English ability better than the baselines and improves both English-to-target and target-to-English question answering, indicating better inner alignment between languages.
  • SALT works with encoder, decoder, and encoder-decoder PLMs as embedding sources, so existing small models can be recycled for language transfer.
  • Replacing the source vocabulary with a target PLM's vocabulary reduces tokenized sequence lengths and total parameters, yielding computational savings during training and inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's success likely depends on vocabulary coverage between the source and target models; the paper's own Arabic results, with the lowest overlap, suggest a testable threshold below which the per-token regression may fail to generalize.
  • Because SALT only changes initialization, its benefits could compound with instruction tuning or longer continued pretraining, a direction the paper leaves open.
  • The same per-token regression logic could be applied to intermediate layers or to contextualized PLM representations, potentially transferring deeper target-language knowledge rather than only static embeddings.
  • Languages without a dedicated target PLM would be out of scope, so a practical extension would need to synthesize or approximate a target embedding space from multilingual PLMs or from the source LLM itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SALT, a cross-lingual transfer method that replaces an English-centric LLM's embedding matrix with a target-language PLM's vocabulary, initializing each non-shared token embedding via a per-token linear least-squares map fitted on Sparsemax-selected shared tokens using fastText similarities. The authors evaluate SALT for German, Arabic, and Vietnamese on knowledge-based benchmarks (ARC, HellaSwag, TruthfulQA), machine reading comprehension (MLQA, XQuAD, Belebele), cross-lingual QA, and language-modeling loss during language-adaptive continual pre-training, using Gemma-2b and XGLM-1.7b as source models and BERT/GPT/T5-based PLMs as target models. They report that SALT outperforms FOCUS, OFA, and multivariate initialization on most averaged benchmarks, converges faster, and better preserves English capability.

Significance. If the empirical claims hold, SALT is a useful and lightweight contribution to cross-lingual LLM transfer: it recycles widely available PLMs, requires no architectural changes to the source LLM beyond embedding and head replacement, and yields both faster convergence and parameter savings. The method is clearly specified, the baselines are appropriate, and the experimental design includes multiple languages, source models, and target architectures, with care taken to use the same external embeddings and tokenizers across methods. The paper also provides honest limitations, including the dependency on available target-language PLMs and the lack of instruction-tuning evaluation. However, the central claim of significant outperformance is not yet backed by statistical evidence, and the key linear-projection step has an unexamined rank-deficiency risk that bears directly on whether the method actually transmits the full representational content of the PLM embeddings.

major comments (3)
  1. [Section 3, Step 4 (Eq. 3)] The least-squares solution X_ti = pinv(E'_ti) E'_si defines a linear map whose rank is at most k, the number of Sparsemax-selected shared anchors. When k < h_t (the target PLM hidden size), applying X_ti to a non-shared PLM embedding first projects that embedding onto the row space of E'_ti and discards the orthogonal complement; the transferred embedding then cannot carry the full representational content of the PLM. The manuscript reports neither the distribution of k after Sparsemax, nor the numerical rank of E'_ti, nor the residuals of the fitted regressions. This is especially pertinent for Arabic with Gemma, where Table 6 shows only 15.6% shared-vocabulary coverage, so a large fraction of non-shared tokens are transferred under conditions where k may be small. Please report these diagnostics and, if rank deficiency is prevalent, discuss or implement a regularized estimator (e.g., ridge) or a lower-rank constrained projection.
  2. [Section 5 (Tables 1-3)] The abstract and Section 5 state that SALT 'significantly outperforms' baselines, but the paper provides no error bars, no multiple-seed results, and no significance tests. The reported averages favor SALT on most language/task combinations, yet individual cells are at parity or below baselines; for example, Table 1 shows XGLM Arabic TruthfulQA with SALT at 28.07 versus FOCUS at 28.72, and Gemma Arabic TruthfulQA with SALT tying Multivariate and FOCUS at 28.98. Without measures of variance, the robustness of the claimed gains and the ranking of methods remain unclear. Please add standard deviations over at least three random seeds for the main tables, and a paired test (e.g., bootstrap over language/task cells) for the aggregate comparisons.
  3. [Section 5.3 (Tables 2 and 3)] The MRC and cross-lingual results are reported only for Gemma as the source model. The paper's broader conclusions that SALT 'preserves English capability' and improves cross-lingual understanding are therefore not verified for XGLM, even though Table 1 suggests the benefits may be smaller or more variable there. Please either add corresponding XGLM results for Tables 2 and 3 or explicitly restrict the cross-lingual and MRC claims to the Gemma setting.
minor comments (4)
  1. [Section 3, Step 1] The handling of target tokens absent from fastText is underspecified: it is unclear whether such tokens receive the random initialization and are then excluded from the regression, or whether they are mapped through a fallback similarity. This matters because the coverage fractions in Table 6 implicitly depend on how these tokens are treated.
  2. [Section 4.4 / Table 5] Table 5 reports evaluation loss at various percentages of total training steps, but the evaluation set (whether the validation split from Appendix C or another corpus) is not specified; please clarify.
  3. [Figure 2] The loss curves are reported without any smoothing or reproducibility details (e.g., which random seed, whether curves are averaged), and Appendix D reports only the final training loss; providing per-step numerical values or a reproducible plotting script would strengthen the convergence claim.
  4. [Limitations] The statement that 'In principle, SALT can also be applied to larger models (>7b)' is speculative and unsupported by experiments; consider softening it or citing evidence from related vocabulary-transfer work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SALT is an embedding initialization method evaluated on held-out downstream benchmarks, and its construction does not reduce to its evaluation inputs.

full rationale

The paper's derivation chain is self-contained as a method proposal: SALT constructs new target-language embeddings by fitting a per-token least-squares map (Eq. 3) from target-PLM shared-token embeddings to source-LLM shared-token embeddings, using nearest-neighbor selection driven by external fastText vectors and Sparsemax. This construction is not fitted to any downstream benchmark result, and the reported gains on ARC, HellaSwag, TruthfulQA, MLQA, XQuAD, and Belebele are evaluated after language-adaptive continual pre-training on an unlabeled corpus. The target PLM embeddings, source LLM embeddings, and fastText vectors are all external inputs that do not encode the evaluation outcomes. There is no self-citation chain invoked as proof, no uniqueness theorem imported from the authors' prior work, and no fitted parameter being renamed as a prediction. The stated limitation that SALT requires a target-language PLM is an applicability constraint, not circularity. The potential rank-deficiency of the least-squares projection when Sparsemax selects few anchors is a correctness or robustness concern, but it does not make the method's derivation equivalent to its inputs by construction. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No numbers were fitted to benchmark outcomes; the free parameters are architectural (k via sparsemax) rather than data-fitted. The main assumptions are about the quality of PLM embeddings, the neutrality of shared tokens, and the linearity of the embedding spaces.

assumptions (6)
  • domain assumption Target-language PLM embeddings contain richer semantic information for the target language than source LLM embeddings.
    Section 3 opens with this assumption; it motivates using PLM embeddings as the source of transfer.
  • domain assumption Shared vocabulary tokens are language-neutral and have been adequately learned by the source LLM, so their embeddings can serve as anchor points.
    Section 3, Step 2 states this, following FOCUS and OFA.
  • domain assumption A linear transformation (least-squares fit) in embedding space can transfer semantic information from PLM space to LLM space for individual tokens.
    Section 3, Step 4, Eq. 3; the paper provides no evidence of linearity, it is a modeling choice.
  • domain assumption fastText subword vectors give reliable semantic similarity between tokens across tokenizer boundaries.
    Section 3, Step 1; used to select nearest shared tokens.
  • domain assumption Post-transfer language-adaptive continual pre-training is sufficient to align the new embeddings with the upper layers.
    Section 3, Step 5; this is standard practice in the cited work.
  • standard math Pseudo-inverse least-squares solution is valid for the embedding mapping.
    Eq. 3; standard linear algebra, cited to Peters and Wilkinson (1970).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer." pith.science (2026). https://pith.science/paper/N6ZPYCYP

@misc{pith2026250510945,
  author       = {Pith},
  title        = {Pith review of: Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N6ZPYCYP}},
  note         = {Machine review of arXiv:2505.10945}
}
read the original abstract

Large Language Models (LLMs) increasingly incorporate multilingual capabilities, fueling the demand to transfer them into target language-specific models. However, most approaches, which blend the source model's embedding by replacing the source vocabulary with the target language-specific vocabulary, may constrain expressive capacity in the target language since the source model is predominantly trained on English data. In this paper, we propose Semantic Aware Linear Transfer (SALT), a novel cross-lingual transfer technique that recycles embeddings from target language Pre-trained Language Models (PLMs) to transmit the deep representational strengths of PLM-derived embedding to LLMs. SALT derives unique regression lines based on the similarity in the overlap of the source and target vocabularies, to handle each non-overlapping token's embedding space. Our extensive experiments show that SALT significantly outperforms other transfer methods and achieves lower loss with accelerating faster convergence during language adaptation. Notably, SALT obtains remarkable performance in cross-lingual understanding setups compared to other methods. Furthermore, we highlight the scalable use of PLMs to enhance the functionality of contemporary LLMs by conducting experiments with varying architectures.

Figures

Figures reproduced from arXiv: 2505.10945 by the authors.

Figure 1
Figure 1. Summary of SALT. By using the paired embeddings of semantically similar tokens for each non-shared [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Causal Language Modeling (CLM) loss of language adaptive continual pre-training. The x-axis represents [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 18 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Arabert: Transformer-based model for arabic language understanding

    Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020, page 9

    work page 2020

  4. [4]

    Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. Aragpt2: Pre-trained transformer for arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196--207

    work page 2021

  5. [5]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. https://doi.org/10.18653/v1/2020.acl-main.421 On the cross-lingual transferability of monolingual representations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.421 2020

  6. [6]

    Abhinand Balachandran. 2023. Tamil-llama: A new tamil language model based on llama 2. arXiv preprint arXiv:2311.05845

    arXiv 2023

  7. [7]

    Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. https://doi.org/10.18653/v1/2024.acl-long.44 The belebele benchmark: a parallel reading comprehension dataset in 122 language variants . In Proceedings of the 62nd Annual Meeting of t...

    doi:10.18653/v1/2024.acl-long.44 2024

  8. [8]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135--146

    2017

Show all 59 references
  1. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  2. [10]

    Branden Chan, Stefan Schweter, and Timo M \"o ller. 2020. https://doi.org/10.18653/v1/2020.coling-main.598 G erman`s next language model . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788--6796, Barcelona, Spain (Online). Internation...

    2020 doi

  3. [11]

    Chau, Lucy H

    Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.118 Parsing with multilingual BERT , a small corpus, and a small treebank . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324--1334, Online. ...

    2020 doi

  4. [12]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    2018 arXiv

  5. [13]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  6. [14]

    Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177

    2023 arXiv

  7. [15]

    dbmdz. 2021. https://huggingface.co/dbmdz/german-gpt2 dbmdz/german-gpt2

    2021

  8. [16]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  9. [17]

    Konstantin Dobler and Gerard De Melo. 2023. Focus: Effective embedding initialization for monolingual specialization of multilingual models. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  10. [18]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  11. [19]

    AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. https://doi.org/10.18653/v1/2023.arabicnlp-1.20 Octopus: A multitask model and toolkit for A rabic natural language generation . In Proceedings of ArabicNLP 2023, pages 232--243, Singapore (Hybrid)...

    2023 doi

  12. [20]

    Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. arXiv preprint arXiv:2404.17790

    2024 arXiv

  13. [21]

    Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. 2022. Fast vocabulary transfer for language model compression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 409--416

    2022

  14. [22]

    Evangelia Gogoulou, Ariel Ekgren, Tim Isbister, and Magnus Sahlgren. 2022. Cross-lingual transfer of monolingual models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 948--955

    2022

  15. [23]

    Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.878 C hat GPT beyond E nglish: Towards a comprehensive evaluation of large language models in multilingual learning ...

    2023 doi

  16. [24]

    Rossi, and Thien Huu Nguyen

    Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023 b . https://arxiv.org/abs/2307.16039 Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedbac...

    2023 arXiv

  17. [25]

    Celio Larcher, Marcos Piau, Paulo Finardi, Pedro Gengo, Piero Esposito, and Vinicius Carid \'a . 2023. Cabrita: closing the gap for foreign languages. arXiv preprint arXiv:2308.11878

    2023 arXiv

  18. [26]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \'e , et al. 2023. Bloom: A 176b-parameter open-access multilingual language model

    2023

  19. [27]

    Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. Mlqa: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315--7330

    2020

  20. [28]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022 a . https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2022 doi

  21. [29]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022 doi

  22. [30]

    Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Sch \"u tze. 2024. Ofa: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1067--1097

    2024

  23. [31]

    Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364

    2019 arXiv

  24. [32]

    Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning, pages 1614--1623. PMLR

    2016

  25. [33]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. Wechsel: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa...

    2022

  26. [34]

    Nandini Mundra, Aditya Khandavally, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M Khapra. 2024. An empirical comparison of vocabulary expansion and initialization approaches for language models. In Proceedings of the 28th Conference on Computational Natural La...

    2024

  27. [35]

    Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.92 P ho BERT : Pre-trained language models for V ietnamese . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037--1042, Online. Association for Computati...

    2020 doi

  28. [36]

    NlpHUST. 2022. https://huggingface.co/NlpHUST/gpt2-vietnamese Nlphust/gpt2-vietnamese

    2022

  29. [37]

    Malte Ostendorff and Georg Rehm. 2023. Efficient language model training through cross-lingual and progressive transfer learning. arXiv preprint arXiv:2301.09626

    2023 arXiv

  30. [38]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  31. [39]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277

    2023 arXiv

  32. [40]

    Gwen Peters and James Hardy Wilkinson. 1970. The least squares problem and pseudo-inverses. The Computer Journal, 13(3):309--316

    1970

  33. [41]

    Long Phan, Hieu Tran, Hieu Nguyen, and Trieu H. Trinh. 2022. https://aclanthology.org/2022.naacl-srw.18 V i T 5: Pretrained text-to-text transformer for V ietnamese language generation . In Proceedings of the 2022 Conference of the North American Chapter of the Association for...

    2022

  34. [42]

    Poorna Chander Reddy Puttaparthi, Soham Sanjay Deo, Hakan Gul, Yiming Tang, Weiyi Shang, and Zhe Yu. 2023. Comprehensive evaluation of chatgpt reliability through multilingual inquiries. arXiv preprint arXiv:2312.10524

    2023 arXiv

  35. [43]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  36. [44]

    Fran c ois Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of llms for low-resource nlp. arXiv preprint arXiv:2408.04303

    2024 arXiv

  37. [45]

    Stefan Schweter, Philip May, and Philipp Schmid. 2024. https://huggingface.co/GermanT5/t5-efficient-gc4-german-base-nl36 Germant5/t5-efficient-gc4-german-base-nl36

    2024

  38. [46]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03...

    2023

  39. [47]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  40. [48]

    Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306

    2020 arXiv

  41. [49]

    Zihan Wang, K Karthikeyan, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual bert to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649--2656

    2020

  42. [50]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022 a . Emergent abilities of large language models. Transactions on Machine Learning Research

    2022

  43. [51]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  44. [52]

    https://dumps.wikimedia.org Wikimedia downloads

    Wikimedia. https://dumps.wikimedia.org Wikimedia downloads

  45. [53]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  46. [55]

    Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024 b . Vocabulary expansion for low-resource cross-lingual transfer. arXiv preprint arXiv:2406.11477

    2024

  47. [56]

    Haotian Ye, Yihong Liu, Chunlan Ma, and Hinrich Sch \"u tze. 2024. https://doi.org/10.18653/v1/2024.insights-1.1 M o SEC ro T : Model stitching with static word embeddings for crosslingual zero-shot transfer . In Proceedings of the Fifth Workshop on Insights from Negative Resu...

    2024 doi

  48. [57]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  49. [58]

    Qingcheng Zeng, Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, Rob Voigt, and Jie Yang. 2023. Greenplm: cross-lingual transfer of monolingual pre-trained language models at almost no cost. In Proceedings of the Thirty-Second International...

    2023

  50. [59]

    Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055

    2024 arXiv

  51. [60]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.