Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery

Fan Jiang; Grace Chung; Honglin Yu; Trevor Cohn

REVIEW 4 major objections 4 minor 17 references

Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Franken-Adapter claims that training only a new embedding layer and custom tokenizer, then attaching it to an instruction-tuned model, gives up to 20% gains across 96 low-resource languages with under 1% English regression.

desk verdict Embedding surgery has real gains for low-resource classification, but the headline 'up to 20%' is not isolated from the LoRA stage's extra multilingual data, and the English regression claim is overstated. read the letter →

arxiv 2502.08037 v1 pith:YEQBCKEW submitted 2025-02-12 cs.CL

Fan Jiang , Honglin Yu , Grace Chung , Trevor Cohn This is my paper · ORCID

classification cs.CL

keywords Franken-Adaptercross-lingualtransferembeddingsurgerytokenizercustomizationlow-resourcelanguageszero-shotinstructiontuningGemma2LoRAadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large language model can be given new languages without retraining the whole network. Its method keeps the transformer body frozen, learns a new multilingual embedding layer plus a custom tokenizer on target-language text, and then attaches that layer to an instruction-tuned copy of the same model for zero-shot cross-lingual transfer. On Gemma2 models up to 27B parameters, the authors report gains of up to 20% across 96 languages on classification, translation, and math reasoning, with English regressions under 1%. The practical significance is that language adaptation becomes a cheap, modular, post-hoc operation that reuses existing skills instead of risking catastrophic forgetting, and the custom tokenizer also speeds up inference.

What carries the argument

The carrying mechanism is embedding surgery: freezing the transformer body and training only the token embedding matrix on multilingual data, based on the assumption that the body holds language-general knowledge while the embedding layer carries language-specific form. The second piece is the Prune-with-Extension vocabulary: prune non-English tokens from the original tokenizer, extend with BPE tokens for the target language group, keep the vocabulary size constant, and initialize new embeddings by copying overlapping token embeddings and averaging subtoken embeddings for new tokens. The third piece is composition: the adapted embeddings are paired with an instruction-tuned transformer body, and LoRA tuning on a mixed multilingual-plus-instruction corpus reconnects the two halves for generative tasks.

What would settle it

Take an English-centric decoder-only model with no multilingual continued pre-training, apply the full Franken-Adapter recipe (pruned-and-extended BPE vocabulary, embedding-only tuning, composition with an instruction-tuned body) on a language group such as SEA, and measure zero-shot FLORES-200 and BELEBELE scores: if the gains over the vanilla model disappear or English scores drop by more than the reported under-1%, the universality assumption is refuted. The paper itself reports exactly this failure for Gemma2-2B without a multilingual warmup, so reproducing it on a second architecture would settle the matter.

Watch

Extended reading notes

Core claim

The central discovery is that embedding tuning suffices for language adaptation of modern decoder-only LLMs, provided the tokenizer treats target languages fairly. The paper constructs a customized vocabulary by pruning the original tokenizer down to English tokens and extending it with BPE tokens trained on low-resource language data, keeping the total vocabulary size fixed. It trains only the embedding matrix on multilingual data while the transformer is frozen, then composes those embeddings with a transformer body instruction-tuned on English alignment data, calling the result Franken-Adapter, with optional LoRA weights to reconcile the two independently trained components. In this setup, language-adapted Gemma2, Aya23, and PaLM2 models beat their pre-trained and instruction-tuned baselines across five multilingual benchmarks, and math-tuned models gain 14% over their base across 20 languages when fitted with the new embeddings.

Load-bearing premise

The load-bearing premise is that the transformer body of a pre-trained LLM already contains universal cross-lingual knowledge and only the embedding layer is language-specific; if the base model has too little multilingual ability to begin with, embedding-only tuning cannot transfer knowledge, and the paper's own Figure 23 shows this failure for Gemma2-2B without a multilingual warmup.

Editorial extensions

If this is right

Embedding-only tuning with customized tokenizers can serve as a low-cost alternative to full continued pre-training for adapting LLMs to many low-resource languages, with no extra inference cost at decode time.
Language-adapted embeddings can be combined with any instruction-tuned version of the same base model, enabling zero-shot cross-lingual transfer without training per language or per task.
Custom vocabularies improve inference efficiency, with throughput gains up to 28% on low-resource languages, as well as improving accuracy.
The approach transfers reasoning skills post hoc: math-tuned and RL-aligned models gain up to 14% over their base across 20 languages when fitted with adapted embeddings.
Franken-Adapter's modularity lets multiple skills share one set of adapted embeddings, avoiding the redundant full-parameter training that continued pre-training baselines require.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if the universality assumption holds, the same surgery might adapt LLMs to non-language domains, such as specialized code or scientific vocabularies, by treating the embedding layer as the only domain-specific surface; the paper hints at this in its impact statement but does not test it.
Beyond the paper: the inverse correlation between tokenizer fertility and downstream score suggests a design rule that tokenizers can be optimized directly for low-resource task performance rather than corpus likelihood, which could be validated by training tokenizers under explicit fertility constraints.
Beyond the paper: because custom vocabularies give the largest relative gains on smaller models, embedding surgery may be a particularly effective lever for making small on-device models serve low-resource languages.
Beyond the paper: the paper leaves open whether adapted embeddings preserve alignment and safety behavior after being attached to instruction-tuned bodies; a targeted safety benchmark in target languages would determine whether the under-1% English regression masks larger alignment regressions elsewhere.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the Franken-Adapter paper. My take: the core method—prune-and-extend tokenizer plus embedding tuning on a frozen transformer—does produce genuine gains for classification and some low-resource tasks, and the tokenizer fertility analysis is worth reading. But the paper's strongest claims are not pinned down. The abstract promises 'up to 20%' and 'minimal regressions (<1%)' in English. The regressions claim doesn't survive contact with Figure 11, which shows consistent drops across all eight English benchmarks, and no aggregate number is given. The 'up to 20%' comes from the LoRA-Adapt rows, and those rows conflate the embedding surgery with extra training on Dmix = Dla ∪ Dit. The paper never runs FLAN+LoRA on Dmix with the original embeddings. That control is the whole ballgame: Figure 19 already shows LoRA gains require both Dla and Dit, and Figure 10 shows the simpler IT+Lang-Adapt beats Franken-Adapter+LoRA at 9B and 27B on the five-task average. So the modular two-stage assembly is not obviously better than just doing embedding tuning on the IT model, and the mechanism credit is unresolved.

What the paper does well: the experiments are broad (Gemma2 2B/9B/27B, Aya23, PaLM2; up to 96 languages), the ablations on tokenizer construction are careful, and the honest reporting of generative-task degradation in Section 3.5.2 is a plus. The Lang-Adapt results on their own are a real contribution; the comparison with full-parameter NTL training is fair, though the compute footnote is sobering: 200B tokens and up to a week on 256 TPUv5s is not 'cost-effective' in any ordinary sense.

On the math-transfer scenario, the paper lacks a control where the math-tuned model gets LoRA on Dmix without the new embeddings. The 14% improvement claim is therefore likewise unisolated.

There's no code or data release, which hurts reproducibility of the tokenizer pipeline. Also, the caveat in Figure 23—that Gemma2-2B fails without multilingual warmup—shows the universal-knowledge assumption is fragile for smaller models, and the paper admits as much.

I would send this to review. The empirical scope and the tokenizer analysis are worth refereeing, and the identification problem in the LoRA stage can be fixed with one control. The authors are clearly serious; the paper is coherent and honest about many of its limitations. I'd like to see a revision that runs FLAN+LoRA on Dmix with original embeddings and reports exact English regressions. After that, it would be a solid accept.

For a reading group, it's a good case study in how framing can outrun an ablation.

Referee Report

4 major / 4 minor

Summary. The paper proposes Franken-Adapter, a modular cross-lingual adaptation method for decoder-only LLMs. It first builds a customized BPE tokenizer by pruning the original vocabulary and extending it with target-language tokens, then trains new embeddings on multilingual data while freezing the transformer body. These embeddings are swapped into an instruction-tuned version of the same base model to obtain zero-shot cross-lingual transfer, optionally followed by LoRA adaptation on a mixture of multilingual and instruction data. Experiments cover Gemma2 2B/9B/27B, PaLM2, and Aya23 across classification, translation, summarization, and mathematical reasoning benchmarks, with a wide set of ablations on tokenizer construction, data mixtures, initialization, and inference efficiency. The central claims are that embedding tuning alone is effective for language adaptation and that the assembled Franken-Adapter improves low-resource language performance with only minimal regressions in English.

Significance. If the mechanism is fully validated, the paper offers a cheap, modular route to post-hoc multilingual adaptation of existing LLMs, reusing one language-adapted embedding set across multiple instruction-tuned or skill-tuned bodies. The empirical scope is unusually broad: 96 languages, three model families, and both discriminative and generative tasks. The paper also ships careful ablations that isolate the contribution of customized tokenizers, embedding initialization, and data mixtures, and it provides a comparison against continued pre-training that quantifies catastrophic forgetting. These strengths make the paper a potentially valuable contribution. The main caveat is that the headline LoRA-based results do not yet isolate the embedding-surgery mechanism from the effect of additional LoRA training on a multilingual-instruction mixture, and the English-regression claim in the abstract is not backed by reported aggregates.

major comments (4)

[§3.1, §3.5.2, Table 3] The LoRA-Adapt rows in Table 3 are trained on Dmix = Dla ∪ 10% Dit as defined in §3.1, while the ∗-FLAN baseline is trained on Dit only. The comparison therefore conflates two interventions: the embedding swap and additional LoRA training on a mixed multilingual-instruction corpus. The paper never reports the control of running LoRA on Dmix with the original embeddings, so the reader cannot attribute the Table 3 gains to embedding surgery. This is load-bearing for the headline 'up to 20%' claim and for Figure 1, and the control should be added.
[Abstract, Appendix B, Figure 11] The abstract claims 'minimal regressions (<1%) in English,' but Figure 11 shows regressions on all eight English tasks for both Franken-Adapter variants across all model sizes, and no aggregate regression number is reported anywhere. The paper should provide the exact average English regression with variance across the three embeddings; as it stands, the <1% claim is unsupported and appears inconsistent with the plotted data.
[§2.2, Figure 23] The load-bearing assumption that 'the pre-trained transformer body encapsulates universal cross-lingual knowledge' is shown by the paper's own Figure 23 to fail for Gemma-2B: without a multilingual continued-pretraining warmup, language adaptation does not succeed. This limits the scope of contribution 1 ('embedding tuning is effective for language adaptation of LLMs') to models with sufficient initial multilingual ability. The main text should state this boundary condition explicitly and qualify the claim.
[§3.5.2, Table 2, Table 3] The paper claims Franken-Adapter consistently improves mathematical reasoning (Table 2) while simultaneously noting in §3.5.2 that Franken-Adapter alone is inconsistent on generative tasks, with red entries in Table 3. The small language counts in GSM8K-NTL (20 languages) make the averaged gains in Table 2 sensitive to a few languages; reporting per-language standard errors or significance tests would strengthen the claim. The paper should also specify which benchmark and metric support the 'up to 20%' figure in the abstract.

minor comments (4)

[Footnote 4, §2.1] Footnote 4 states that '440% of the tokens are discarded'; this should be '40%' given the claim of over 60% token overlap in the final vocabulary.
[Appendix E, Figure 15] The caption of Figure 15 says the tokenization comparison is 'on Gemma2,' but the figure shows PaLM2 and Aya23; the caption should be corrected.
[Table 3] The 'Avg.' column in Table 3 should explicitly state whether English scores are included in the average; the text says English is excluded, but the table layout places an 'ENGLISH' column immediately before the Avg column, which is confusing.
[Figure 2] The normalization procedure for Figure 2 (baseline model, metric, and aggregation across benchmarks) is not fully specified in the main text or caption; please clarify how 'Normalized Performance' is computed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the claimed gains are measured on external benchmarks, and no fitted parameter or self-citation reduces the predictions to the method's own inputs.

full rationale

The derivation chain is empirical end-to-end. Embedding tuning trains on Dla (NTL plus Wikipedia/mC4) and is evaluated on external benchmarks (BELEBELE, SIB-200, FLORES-200, XORQA-IN, XSUM-IN, GSM8K-NTL) that are not used as supervision for the reported predictions; no target-task label or evaluation metric is used as a training signal. The LoRA stage trains on Dmix = Dla ∪ 10% Dit, but its gains are measured on held-out benchmarks rather than on Dmix itself, so the result is not an identity or a fitted parameter renamed as a prediction. The tokenizer-fertility analysis is a post-hoc correlation study: the paper explicitly varies tokenizer fertility and measures downstream performance, reporting a correlation rather than deriving benchmark scores from a fitted fertility model. The central assumption that the transformer body encodes universal cross-lingual knowledge while the embedding layer is language-specific is cited from independent prior work (Zhao et al., Wendler et al., Tang et al.) and is not a self-citation; moreover, the paper includes a falsification check (Figure 23) showing the assumption fails for Gemma2-2B without warmup. No equation in the paper defines an output in terms of its own input. The only notable caveat is an experimental confound between the embedding swap and the additional LoRA training on Dmix, but a confound is a correctness risk rather than circularity, and the FA-without-LoRA rows show genuine classification gains from the embedding surgery itself. Therefore no circular steps are identified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumption that the transformer body is language-agnostic and that customized tokenizers can be trained with a hand-selected data mixture. The free parameters are design choices, not fitted to the evaluation benchmarks. No new physical or abstract entities are introduced.

free parameters (6)

Max sampled lines per language = 500,000
Chosen in §2.1 to balance resource levels; claimed to outperform temperature sampling.
Dla sentence/document ratio = 65/35
Hand-set mixture in §3.1; ablations (Figure 18) show both components matter.
UniMax sampling N = 5
Sampling exponent used to up-sample low-resource languages in §3.1.
LoRA rank = 64
Rank used for LoRA-Adaptation in Appendix A.1; adds <1% parameters.
Embedding training tokens = 200B
Total tokens used to train embeddings per Appendix A.1; not a tunable hyperparameter but a critical compute burden.
Learning rates = 1e-4 (PaLM2), 1e-5 (Gemma2, Aya23)
Set in Appendix A.1 per model family.

assumptions (5)

domain assumption The transformer body of a pretrained LLM contains universal cross-lingual knowledge; the embedding layer is language-specific.
Invoked in §2.2 to justify training only embeddings.
domain assumption Reusing English tokens preserves the model's pretrained knowledge.
Stated in §2.1; supported by the Scratch vs Prune+Extension ablation (Figure 9).
ad hoc to paper A BPE tokenizer trained on up to 500K lines per language yields a fair representation for low-resource languages.
Design choice in §2.1; validated empirically by fertility and performance correlations.
domain assumption LoRA adaptation on Dmix reconciles the mismatch between the new embeddings and the instruction-tuned body.
Empirical claim in §2.2; required for generative tasks, no theoretical guarantee.
ad hoc to paper Averaging subtoken embeddings is a sufficient initialization for new tokens.
Used in §2.2; Figure 20 shows it is better than max pooling and random init.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery." pith.science (2026). https://pith.science/paper/YEQBCKEW

@misc{pith2026250208037,
  author       = {Pith},
  title        = {Pith review of: Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YEQBCKEW}},
  note         = {Machine review of arXiv:2502.08037}
}

abstract

The capabilities of Large Language Models (LLMs) in low-resource languages lag far behind those in English, making their universal accessibility a significant challenge. To alleviate this, we present $\textit{Franken-Adapter}$, a modular language adaptation approach for decoder-only LLMs with embedding surgery. Our method begins by creating customized vocabularies for target languages and performing language adaptation through embedding tuning on multilingual data. These pre-trained embeddings are subsequently integrated with LLMs that have been instruction-tuned on English alignment data to enable zero-shot cross-lingual transfer. Our experiments on $\texttt{Gemma2}$ models with up to 27B parameters demonstrate improvements of up to 20% across 96 languages, spanning both discriminative and generative tasks, with minimal regressions ($<$1%) in English. Further in-depth analysis reveals the critical role of customizing tokenizers in enhancing language adaptation, while boosting inference efficiency. Additionally, we show the versatility of our method by achieving a 14% improvement over a math-optimized LLM across 20 languages, offering a modular solution to transfer reasoning abilities across languages post hoc.

Figures

Figures reproduced from arXiv: 2502.08037 by the authors.

**Figure 1.** Zero-shot performance comparison between our best model (Gemma2-27B-Franken-Adapter-LoRA) and stateof-the-art LLMs on five benchmarks. BELEBELE (#59 Langs.) FLORES-200 (#86 Langs.) GSM8K-NTL (#20 Langs.) 0.8 1.0 1.2 1.4 1.6 Normalized Performance Pre-trained LLMs + Emb Surgery BELEBELE (#59 Langs.) SIB-200 (#96 Langs.) FLORES-200 (#86 Langs.) XORQA-IN (#29 Langs.) CROSSSUM-IN (#30 Langs.) 0.8 1.0 1.2 1.4 1.6 Normal… view at source ↗

**Figure 2.** Result summary across diverse benchmarks. Scores are normalized v.s. Pre-trained (top) and instruction-tuned (bottom) LLMs. All scores are macro-averaged across all sizes of Gemma2. reduces their utility for users speaking under-represented languages (Ahia et al., 2023). A widely adopted approach to multilingual adaptation involves continued pre-training on additional data in target languages by using pre-trained L… view at source ↗

**Figure 3.** Overview of our Franken-Adapter pipeline: 1) pre-train a LLM on English-dominant data; 2a) freeze the original embeddings of LLMs and instruction-tune the transformer body using English alignment data; 2b) learn new multilingual embeddings by freezing the transformer body for target language adaptation of LLMs; 3) combine new embeddings with instruction-tuned transformer body as the Franken-Adapter and further perfo… view at source ↗

Figures from the paper (14 more)

**Figure 4.** Figure 4: The tokenization comparison between using the vanilla and customized multilingual tokenizers on Gemma2. Tok. Length refers to the average number of tokens required to represent the same amount of texts. only English tokens before adding those from low-resource language…

**Figure 5.** Figure 5: shows the absolute gains of language adaptation on Gemma2 models. When evaluating across three language groups, we observe that language adaptation consistently outperforms vanilla Gemma2 models, demonstrating a significant performance advantage. The performance gains…

**Figure 6.** Figure 6: Ablations on tokenizers for language adaptation. Macroaveraged scores on SEA subset of FLORES-200 and BELEBELE are reported. ∆T F : % tokenizer fertility reduction. this phenomenon to intrinsic difficulty: classification tasks are generally easier as the solution spac…

**Figure 7.** Figure 7: Franken-Adapter result comparison when using original and customized tokenizers. The macro-averaged scores on SEA subset of BELEBELE, SIB-200 and FLORES-200 are reported [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 9.** Figure 9: Ablations on tokenizer building methods. We report SEA language adaptation on Gemma2-2B. 2 9 27 #Parameters (B) 30 40 50 Averaged Score IT IT-Franken-Adapter IT-Franken-Adapter+LoRA IT+Lang-Adapt [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 10.** Figure 10: Comparing applying language adaptation and FrankenAdapter on Gemma2-IT models. The averaged scores in five evaluation benchmarks ( [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

**Figure 11.** Figure 11: Performance on English tasks. For Franken-Adapter methods, we present the averaged performance with variances across three embeddings (i.e., SEA, AFR, IND). Shading indicates the standard deviations measured over three embeddings. (A), (B)) and select the option with …

**Figure 12.** Figure 12: Language Adaptation on PaLM2 (XXS, S) and Aya23 (8B, 35B). Absolute gains over the pre-trained models are reported. to the FLAN model, although the performance gap closes as model capacity scales up. We believe that these minor regressions are justifiable in light of …

**Figure 13.** Figure 13: The difference between our Franken-Adapter and the continued pre-training (CPT) baseline for zero-shot cross-lingual transfer. The same customized tokenizers are used by CPT. For M language groups and N target skills to be adapted, our Franken-Adapter avoids redundant…

**Figure 14.** Figure 14: The performance comparison between Franken-Adapter (w/ LoRA-Adaptation) and the continued pre-training (CPT) baseline. For English, we report the average across 8 datasets. For the rest of multilingual datasets, we report scores of the SEA subsets. tility using custo…

**Figure 16.** Figure 16: Performance comparison between employing distinct tokenizers for each language group and using a single tokenizer for all language groups (i.e., Joint Tokenizer). The averaged performance of five tasks included in [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

**Figure 17.** Figure 17: Correlation between the performance of language adaptation on PaLM2-XXS with tokenizer fertility. Normalized ChrF++ on FLORES-SEA are reported. ♦ and ★ indicate the original and customized tokenizers in PaLM2. This is followed by applying the standard Franken-Adapter…

**Figure 20.** Figure 20: Ablations on embedding initialization methods. FLORES-SEA language performance of language adaptation on Gemma2-2B is reported. Max Pooling: for each new token in the customized vocabulary, we use the original tokenizer to tokenize it and apply max pooling over the e…

**Figure 21.** Figure 21: The effects of using customized vocabularies with different proportions of tuned parameters. The averaged score on SEA languages of FLORES-200 is reported. LLM to achieve the best overall results. Employing the original embeddings for initialization is essential to l…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 6 canonical work pages

[1]

Pretrain MATH Emb LLM AFR Emb LLM IND Emb FLAN FLAN IND Emb FLAN AFR Emb SEA Emb IND Emb AFR Emb MATH MATH MATH LLM Emb

work page
[2]

Language AdaptationCopy SEA Emb SEA LLM SEA Emb AFR LLM AFR Emb AFR LLM AFR Emb IND LLM IND Emb IND LLM IND Emb

work page
[3]

The difference between our Franken-Adapter and the continued pre-training (CPT) baseline for zero-shot cross-lingual transfer

Instruction Tuning SEA LLM Initialize FLAN MATH FLAN MATH FLAN MATH Franken-Adapter Continued Pre-training Freeze Trainable Figure 13. The difference between our Franken-Adapter and the continued pre-training (CPT) baseline for zero-shot cross-lingual transfer. The same customized tokenizers are used by CPT. For M language groups and N target skills to be...

work page 2023
[8]

doi: 10.18653/v1/2020.emnlp-main.617

Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.617. URL https:// aclanthology.org/2020.emnlp-main.617. Pfeiffer, J., Vuli´c, I., Gurevych, I., and Ruder, S. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10186...

doi:10.18653/v1/2020.emnlp-main.617 2020
[11]

Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y ., Wang, Z., Ebrahimi, S., and Wang, H

URL https://openreview.net/forum? id=fR3wGCk-IXp. Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y ., Wang, Z., Ebrahimi, S., and Wang, H. Continual learning of large language models: A comprehensive survey, 2024. URL https://arxiv.org/abs/2404.16789. Singh, H., Gupta, N., Bharadwaj, S., Tewari, D., and Taluk- dar, P. IndicGenBench: A multilingual be...

arXiv 2024
[12]

URL https://openreview.net/forum? id=6ruVLB727MC. 12 Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery Üstün, A., Aryabumi, V ., Yong, Z., Ko, W.-Y ., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid, A., Vargus, F., Blunsom, P., Longpre, S., Muennighoff, N., Fadaee, M., Kreutzer, J., and Hooker, S. Aya model: An i...

arXiv 2024
[13]

URL https://aclanthology.org/2022

Association for Machine Translation in the Amer- icas. URL https://aclanthology.org/2022. amta-research.8. Zhao, J., Zhang, Z., Gao, L., Zhang, Q., Gui, T., and Huang, X. Llama beyond english: An empirical study on language capability transfer, 2024a. URL https: //arxiv.org/abs/2401.01055. Zhao, Y ., Zhang, W., Chen, G., Kawaguchi, K., and Bing, L. How do...

arXiv 2023
[15]

Pretrain SEA LLM SEA Emb AFR LLM AFR Emb IND LLM IND Emb

work page

Show all 17 references

[74]

findings-acl.74

URL https://aclanthology.org/2021. findings-acl.74. Dua, D., Wang, Y ., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. DROP: A reading comprehension 10 Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery benchmark requiring discrete reasoning over par...

2021 arXiv
[338]

findings-acl.338

URL https://aclanthology.org/2023. findings-acl.338. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Bi- derman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., Tang, X., Radev, D., Aji, A. F., Almubarak, K., Albanie, S., Alyafeai, Z., Webson, A., Ra...

2023
[579]

coling-main.579

URL https://aclanthology.org/2020. coling-main.579. Chen, P., Ji, S., Bogoychev, N., Kutuzov, A., Haddow, B., and Heafield, K. Monolingual or multilingual instruction tuning: Which makes a better alpaca. In Graham, Y . and Purver, M. (eds.), Findings of the Association for Com...

2020 arXiv
[614]

emnlp-main.614

URL https://aclanthology.org/2023. emnlp-main.614. Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Ahmed, M., Bali, K., and Sitaram, S. MEGA: Multilingual evaluation of generative AI. In Bouamor, H., Pino, J., and Bali, K. (e...

2023 arXiv
[800]

emnlp-main.800

URL https://aclanthology.org/2021. emnlp-main.800. Pfeiffer, J., Goyal, N., Lin, X., Li, X., Cross, J., Riedel, S., and Artetxe, M. Lifting the curse of multilinguality by pre- training modular transformers. InProceedings of the 2022 Conference of the North American Chapter of...

2021
[891]

acl-long.891

URL https://aclanthology.org/2023. acl-long.891. Pfeiffer, J., Vuli´c, I., Gurevych, I., and Ruder, S. MAD- X: An Adapter-Based Framework for Multi-Task Cross- Lingual Transfer. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNL...

2023
[2020]

cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper. pdf. Caswell, I., Breiner, T., van Esch, D., and Bapna, A. Lan- guage ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedin...

2020 doi
[2022]

doi: 10.18653/v1/2022.naacl-main.255

Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.255. URL https:// aclanthology.org/2022.naacl-main.255. Riviere, M. et al. Gemma 2: Improving open language models at a practical size, 2024. Robinson, N., Ogayo, P., Mortensen, D. R., and Neubig, G. C...

2022 doi
[2023]

doi: 10.18653/v1/2023.emnlp-main.813

Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.813. URL https:// aclanthology.org/2023.emnlp-main.813. Liu, H., Tam, D., Mohammed, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. Few-shot parameter-efficient fine-tuning is better and cheaper t...

2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Pretrain MATH Emb LLM AFR Emb LLM IND Emb FLAN FLAN IND Emb FLAN AFR Emb SEA Emb IND Emb AFR Emb MATH MATH MATH LLM Emb

work page

[2] [2]

Language AdaptationCopy SEA Emb SEA LLM SEA Emb AFR LLM AFR Emb AFR LLM AFR Emb IND LLM IND Emb IND LLM IND Emb

work page

[3] [3]

The difference between our Franken-Adapter and the continued pre-training (CPT) baseline for zero-shot cross-lingual transfer

Instruction Tuning SEA LLM Initialize FLAN MATH FLAN MATH FLAN MATH Franken-Adapter Continued Pre-training Freeze Trainable Figure 13. The difference between our Franken-Adapter and the continued pre-training (CPT) baseline for zero-shot cross-lingual transfer. The same customized tokenizers are used by CPT. For M language groups and N target skills to be...

work page 2023

[4] [8]

doi: 10.18653/v1/2020.emnlp-main.617

Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.617. URL https:// aclanthology.org/2020.emnlp-main.617. Pfeiffer, J., Vuli´c, I., Gurevych, I., and Ruder, S. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10186...

doi:10.18653/v1/2020.emnlp-main.617 2020

[5] [11]

Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y ., Wang, Z., Ebrahimi, S., and Wang, H

URL https://openreview.net/forum? id=fR3wGCk-IXp. Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y ., Wang, Z., Ebrahimi, S., and Wang, H. Continual learning of large language models: A comprehensive survey, 2024. URL https://arxiv.org/abs/2404.16789. Singh, H., Gupta, N., Bharadwaj, S., Tewari, D., and Taluk- dar, P. IndicGenBench: A multilingual be...

arXiv 2024

[6] [12]

URL https://openreview.net/forum? id=6ruVLB727MC. 12 Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery Üstün, A., Aryabumi, V ., Yong, Z., Ko, W.-Y ., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid, A., Vargus, F., Blunsom, P., Longpre, S., Muennighoff, N., Fadaee, M., Kreutzer, J., and Hooker, S. Aya model: An i...

arXiv 2024

[7] [13]

URL https://aclanthology.org/2022

Association for Machine Translation in the Amer- icas. URL https://aclanthology.org/2022. amta-research.8. Zhao, J., Zhang, Z., Gao, L., Zhang, Q., Gui, T., and Huang, X. Llama beyond english: An empirical study on language capability transfer, 2024a. URL https: //arxiv.org/abs/2401.01055. Zhao, Y ., Zhang, W., Chen, G., Kawaguchi, K., and Bing, L. How do...

arXiv 2023

[8] [15]

Pretrain SEA LLM SEA Emb AFR LLM AFR Emb IND LLM IND Emb

work page