Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new method, SAVA, transplants an Italian tokenizer into English LLMs, cutting Italian token use by 25% and restoring performance after 12B training tokens.

desk verdict Solid empirical recipe for Italian vocabulary adaptation; SAVA is a reasonable new heuristic, but its advantage over simpler FVT is thin and the extrapolation step is under-validated. read the letter →

arxiv 2504.17025 v1 pith:2P4NKKFR submitted 2025-04-23 cs.CL

Luca Moroni , Giovanni Puccetti , Pere-Lluis Huguet Cabot , Andrei Stefan Bejgu , Edoardo Barba , Alessio Miaschi , Felice Dell'Orletta , Andrea Esuli
show 1 more author
Roberto Navigli
This is my paper · ORCID
classification cs.CL
keywords SAVASemanticAlignmentVocabularyAdaptationtokenfertilityItalianlanguagemodelscontinualtrainingembeddinginitializationtokenizersubstitution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

English-heavy LLMs encode Italian inefficiently: they split words into more tokens than an Italian-native tokenizer would, and pay for it in slower inference, larger memory, and a bigger vocabulary than the language needs. This paper proposes Semantic Alignment Vocabulary Adaptation (SAVA), which replaces the English tokenizer and vocabulary of Mistral-7B-v0.1 and Llama-3.1-8B with Minerva-LLMs' Italian tokenizer, and initializes the new embeddings with an affine map $\phi(x)=Wx+b$ learned from the tokens the two vocabularies share. The reported result is that token fertility on Italian text falls by 25% for Mistral and 16% for Llama, Llama loses about one billion parameters, and after about 12 billion tokens of continual training on mixed Italian-English data the adapted models match or approach the base models and a language-adaptive pretraining baseline on Italian benchmarks. If the paper is right, SAVA is a cheaper route to Italian-capable LLMs than continued pretraining with the original English tokenizer, with faster convergence and lower inference cost.

What carries the argument

The central mechanism is Semantic Alignment Vocabulary Adaptation (SAVA), an embedding-initialization procedure built on the premise that two LLM embedding spaces are related by an affine map. For every token $t_i$ in the shared vocabulary $V_s \cap V_t$, SAVA has two representations: $E^{t_i}_s$ in the English model and $E^{t_i}_h$ in the helper model Minerva-3B. It fits a single-layer feed-forward network $\phi(x)=Wx+b$ by minimizing the squared error between $W E^{t_i}_h + b$ and $E^{t_i}_s$ over the shared tokens, then initializes each Italian-only token $t_i \in V_t \setminus V_s$ as $\phi(E^{t_i}_h)$. This replaces bilingual-dictionary or averaged-sub-token heuristics with one learned projection, and the paper's loss curves and downstream scores attribute SAVA's faster convergence and closer structural alignment to the helper model to this initialization. Fertility, the average number of tokens into which a word is split, is the efficiency quantity that motivates the whole substitution.

What would settle it

Apply SAVA to a target language whose subword vocabulary barely overlaps with the source model's (e.g., Finnish), using the same 12B-token continual-training budget. If the out-of-vocabulary embeddings produced by the shared-token map do not lower the initial training loss or final downstream scores relative to FVT and random initialization, the assumption that the linear relation extrapolates to unseen subwords is wrong.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that vocabulary substitution for decoder-only LLMs can be made data-efficient by aligning embedding spaces before training. SAVA keeps the source model's weights, swaps the tokenizer and vocabulary for Minerva-LLMs' Italian tokenizer, and initializes every new Italian token's embedding by applying the learned map $\phi$ to that token's representation in Minerva-3B. The affine map is trained with mean squared error on the 16,438 tokens that Mistral and Minerva share (20,358 for Llama), and the same map is applied to the Italian-only tokens the source model never saw. After about 12 billion tokens of continual training on 75% Italian and 25% English text, the adapted models match or approach the scores of LAPT and, on several tasks, the unmodified base models, while SAVA's training loss is lower from the outset and its embedding structure is closer to Minerva-3B than any other heuristic tested.

Load-bearing premise

SAVA's success rests on the assumption that the affine relationship learned from the tokens the English and Italian models share also holds for Italian subwords that exist only in the new vocabulary.

Editorial extensions

If this is right

  • Italian text is encoded with 25% fewer tokens for Mistral-7B and 16% fewer for Llama-3.1-8B, which translates directly into shorter inference time and lower serving cost.
  • The Llama-3.1-8B adaptation removes roughly 75% of the original vocabulary and about one billion parameters, so the resulting model is 10% smaller and faster to load.
  • SAVA-adapted models converge faster during continual training, matching at 400 batches what random initialization reaches only at the end of the 12B-token budget.
  • After continual training, vocabulary-adapted models reach scores comparable to LAPT on the Italian multiple-choice benchmarks and generative tasks, while retaining slightly better English performance among the adapted variants.
  • Mistral's SAVA adaptation reaches base-model performance after only about 2 billion tokens, indicating the 12B-token budget is not a hard requirement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper demonstrates SAVA only for Italian, but the mechanism is language-agnostic; the same linear-map recipe should work for any target language that has a native tokenizer and a smaller helper model.
  • The ablations show that helper-model size (350M–3B) changes the loss trajectory less than the number of shared tokens, suggesting that even a small, well-matched helper tokenizer could suffice for low-resource languages.
  • Because the vocabulary replacement alone removes a large share of Llama's embedding parameters, the technique could be combined with quantization or layer pruning to obtain memory savings well beyond the reported 10%.
  • A stronger helper model than Minerva-3B could improve SAVA further; the paper notes Minerva-7B was not available at the time of writing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses the inefficiency of English-centric LLM tokenizers on Italian by comparing vocabulary adaptation methods and introducing SAVA, which replaces the source tokenizer with Minerva's Italian tokenizer and initializes target embeddings by learning a linear map from Minerva-3B's embedding space to the source model's embedding space. The authors apply the methods to Mistral-7B-v0.1 and Llama-3.1-8B, report fertility reductions of 25% and 16%, and a 1B-parameter reduction for Llama, then continue training on 12B tokens of mixed Italian/English data. They evaluate on translated multiple-choice benchmarks (ITA-Bench), FLoRes, and SQuAD-it, and analyze the resulting embedding structure. The central claim is that SAVA-initialized models recover base-model performance after about 12B tokens and converge faster than other initialization heuristics.

Significance. The paper's empirical scope is a strength: two 7-8B models, a fixed 12B-token budget, a consistent evaluation harness, publicly released code and models, and an environmental-impact statement. If the results hold, vocabulary substitution plus continual training is a practical alternative to full Italian continual pretraining. The paper also provides a useful comparison of FVT, CLP, Random, and LAPT. However, the incremental advantage of SAVA over the much simpler FVT is small and not statistically established, and the extrapolation at the core of SAVA is untested. The empirical recovery claim is well supported; the novel-method claim needs strengthening.

major comments (4)
  1. [Section 3.1, Eq. (1)] The load-bearing assumption of SAVA is that the linear map phi learned on the shared vocabulary Vs intersection Vt transfers to all tokens in Vt without Vs, since those tokens are initialized solely through phi. The paper does not test this extrapolation directly. Appendix B varies the amount of training data and the helper model size but only measures the continual-training loss, and Section 6 uses 256 anchor tokens and reports an aggregate relative-representation similarity, which does not isolate the quality of the out-of-vocabulary token embeddings. Because the shared tokens may be systematically different from Italian-only subwords (e.g., punctuation, ASCII fragments, and frequent English tokens), the claim that SAVA provides semantically aligned initializations for Italian-specific tokens remains unverified. I recommend adding a diagnostic: measure the mapping error on a held-out split of shared tokens, or compare nearest neighbors or zero-shot perplexity restricted to the Vt without Vs tokens against an independent bilingual lexicon.
  2. [Tables 3 and 4; Section 5.1.1] The paper's central methodological contribution, SAVA, is not statistically distinguishable from FVT in the reported results. In Table 3 at 2000 steps, SAVA averages 66.99 versus 66.94 for FVT, and in Table 4, SAVA averages 66.29 versus 65.79 for FVT; per-task standard errors are roughly 0.4 to 1.2 points, and no multiple seeds or significance tests are reported. The text nonetheless claims that SAVA consistently achieves higher overall scores and faster convergence (Figures 2 and 3). Without a significance test, multiple independent runs, or a larger effect, the comparison only supports the weaker claim that SAVA is competitive with FVT. The conclusions and abstract should be reworded accordingly, or the missing evidence should be added.
  3. [Abstract and Introduction; Table 1] The statement that for Mistral-7B-v0.1 'we do not increase its vocabulary size or model parameters' is contradicted by Table 1, which reports 32,768 tokens versus 32,000 and 7.25B parameters versus 7.24B. This is a stated contribution and should be corrected, or the wording should be changed to describe a negligible increase rather than no increase.
  4. [Section 5.1.2; Table 11] The claim that 'for both models the SAVA approach leads the model to achieve slightly higher performance in the source language' is not supported for Llama-3.1-8B: Table 11 shows SAVA at 75.33 average versus LAPT at 77.29 at 3000 steps, so LAPT is substantially higher. The sentence should be qualified to refer only to the vocabulary-adapted methods, or removed.
minor comments (5)
  1. [Section 5.1.1 and Section 7] The conclusion that 'Mistral-7B-v0.1 reached base model performance after processing 2 billion tokens' is not tied to a specific table; the 400-batch checkpoint (approximately 2.5B tokens) is not reported in the tables, so please add the corresponding values or cite the relevant figure more precisely.
  2. [Table 2 and Section 4.1] The text says the Minerva tokenizer has 'on average 25% of fertility gain' compared to Mistral on Italian text, but the two reductions in Table 2 are about 26% on CulturaX and 19% on Wikipedia; please state the exact computation or report the numbers as 'up to 26%'.
  3. [Tables 2 and 4] The model name is written inconsistently as 'LLaMa-3-8B' in Table 2 and 'Llama-3.1-8B' elsewhere; please use one consistent name throughout.
  4. [Section 5.1.1] The sentence 'SAVA and FVT achieve results at 400 batches that are comparable to those of the Random approach at the end of training' refers to a checkpoint that does not appear in the tables; please include the 400-batch numbers or state the corresponding token count explicitly.
  5. [Section 6] The description of the anchor selection as '128 non-prefix tokens and 128 prefix tokens' is ambiguous, since 'prefix' could mean tokens that begin a word rather than model prefix tokens; please clarify the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SAVA's linear mapping is an extrapolation heuristic, and the paper's main downstream claims are externally evaluated on translated standard benchmarks.

full rationale

The paper's central claim is that vocabulary substitution with Minerva's tokenizer, SAVA initialization, and roughly 12B tokens of continual training lets English-centric LLMs recover performance on Italian tasks while reducing fertility and model size. The fertility and parameter reductions are direct properties of the chosen tokenizer substitution, not derived predictions. SAVA's core step (Section 3.1, Eq. 1) fits a linear map phi on tokens in Vs∩Vt and applies it to tokens in Vt\Vs; this is an extrapolation assumption and a legitimate modeling choice, not a circular reduction, because the downstream evaluations are measured on ITA-Bench translations of MMLU, HellaSwag, ARC, PIQA, SciQ, BoolQ, FLoRes, and SQuAD-it, which are independent of the fitted mapping. The paper's Table 7 similarity-to-Minerva analysis is admittedly expected for SAVA and CLP because both methods leverage Minerva-3B's embedding space, and the paper does not use that similarity as the primary evidence for its performance claims; it explicitly notes the outcome is 'to be expected' and relies on downstream task scores for its conclusions. The same-group resources (Minerva-LLMs, ITA-Bench) are self-citations, but they are not load-bearing in a circular sense: Minerva is an externally trained model family, and ITA-Bench consists of translated standard benchmarks whose evaluation is falsifiable outside the paper's fitted values. The acknowledged limitation about automatically translated benchmarks introduces noise, but it is a data-quality concern, not circularity. Overall, the derivation chain is self-contained and empirically grounded.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the helper model Minerva-3B (same-group), the computed token budget, the data mix, and the assumed linearity of embedding spaces. No new physical entities are introduced.

free parameters (4)
  • Continual training token budget = 12 billion tokens
    Stopping threshold set by compute constraints, not by convergence; all methods compared at the same budget so relative comparison is fair, but absolute claims of limited training depend on this choice.
  • Italian/English data ratio = 75% Italian, 25% English
    Adopted from Csaki et al. 2024; not ablated here, and it directly controls the trade-off between Italian gains and English forgetting.
  • Continual training learning rate = 1e-5
    Fixed across all runs; stated to be the default in the training library, so its adequacy for each adaptation method is not verified.
  • Mapping training steps and learning rate = 1000 steps, lr 1e-3
    Used to fit the linear map phi in SAVA; from the latentis library, not tuned per model.
assumptions (4)
  • domain assumption Embedding spaces of different models are related by a linear (conformal) map
    Invoked in Section 3.1 and grounded in Maiorca et al. 2024 and Moschella et al. 2023; the paper does not re-derive or test this on Italian subwords.
  • ad hoc to paper The linear map phi trained on shared tokens generalizes to out-of-vocabulary tokens
    Central to SAVA: phi is trained only on the shared vocabulary but applied to the out-of-vocabulary tokens in Eq. (1); no supervision exists for Italian-only tokens.
  • domain assumption Automatically translated benchmarks preserve Italian capability signal
    ITA-Bench is generated by Tower-Instruct-v0.2; the authors acknowledge in Section 8 that translation noise may obscure abilities.
  • domain assumption Minerva tokenizer is a suitable target tokenizer for Italian
    Taken from Minerva-LLMs (Orlando et al. 2024), which is trained from scratch on CulturaX; the paper does not train or justify its own target tokenizer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation." pith.science (2026). https://pith.science/paper/2P4NKKFR

@misc{pith2026250417025,
  author       = {Pith},
  title        = {Pith review of: Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2P4NKKFR}},
  note         = {Machine review of arXiv:2504.17025}
}
read the original abstract

The number of pretrained Large Language Models (LLMs) is increasing steadily, though the majority are designed predominantly for the English language. While state-of-the-art LLMs can handle other languages, due to language contamination or some degree of multilingual pretraining data, they are not optimized for non-English languages, leading to inefficient encoding (high token "fertility") and slower inference speed. In this work, we thoroughly compare a variety of vocabulary adaptation techniques for optimizing English LLMs for the Italian language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a novel method that leverages neural mapping for vocabulary substitution. SAVA achieves competitive performance across multiple downstream tasks, enhancing grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and reducing the number of parameters by 1 billion. We show that, following the adaptation of the vocabulary, these models can recover their performance with a relatively limited stage of continual training on the target language. Finally, we test the capabilities of the adapted models on various multi-choice and generative tasks.

Figures

Figures reproduced from arXiv: 2504.17025 by the authors.

Figure 1
Figure 1. Fertility for two different tokenizers, Mistral￾7B-v0.1 (left) and Minerva (right), over Italian texts from CulturaX (blue) and Wikipedia (red). in multiple target languages – have been released over the years (Le Scao et al., 2023), they still fall short of achieving performance levels comparable to models pre-trained in English. The primary chal￾lenge is addressing underrepresented languages, where large, clean, o… view at source ↗
Figure 2
Figure 2. Average performance of Mistral-7B-v0.1 based models during training on Italian translated benchmarks. The average was calculated over six datasets. generative benchmarks, specifically FLoRes and SQuAD-it. In this and subsequent sections we indi￾cate the continual training of the base model with￾out vocabulary adaptation by the LAPT acronym. 5.1 Multi-choice Setting 5.1.1 Italian Results We report results on Italian … view at source ↗
Figure 5
Figure 5. reports the average scores of Llama-3.1- 8B models on English benchmarks during training. In this setting, LAPT maintains higher performance on average; intuitively, this could be attributed to the larger vocabulary of Llama-3.1-8B (75% big￾ger), which enables better performance during lan￾guage adaptation, avoiding catastrophic forgetting of the source language [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Average performance of Mistral-7B-v0.1 based models during training on English benchmarks. The average was calculated over six datasets. erage score at the end of the adaptation process. All the adapted models diminish in performance compared to the base one in the Eng…
Figure 6
Figure 6. Figure 6: Loss during continual training of Mistral-7B￾v0.1 models [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Loss during continual training of Llama-3.1- 8B models. when compared to the CLP and Random models. Notably, CLP appears to lag behind Random ini￾tially. Looking at Llama-3.1-8B losses, in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Similarity across models after continual train [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Loss during continual training of Mistral mod [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Loss during continual training of Mistral [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 6 canonical work pages

  1. [1]

    Pierpaolo Basile, Elio Musacchio, Marco Polignano, Lucia Siciliani, Giuseppe Fiameni, and Giovanni Semeraro. 2023. https://arxiv.org/abs/2312.09993 Llamantino: Llama 2 models for effective text generation in italian language . Preprint, arXiv:2312.09993

    arXiv 2023

  2. [2]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

    2020

  3. [3]

    Brown, Stephen A

    Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. https://aclanthology.org/J93-2003 The mathematics of statistical machine translation: Parameter estimation . Computational Linguistics, 19(2):263--311

    1993

  4. [4]

    Chau, Lucy H

    Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.118 Parsing with multilingual BERT , a small corpus, and a small treebank . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324--1334, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.118 2020

  5. [5]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language...

    doi:10.18653/v1/n19-1300 2019

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    arXiv 2018

  7. [7]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2020.acl-main.747 2020

  8. [8]

    Costa - juss \` a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y

    Marta R. Costa - juss \` a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y. Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Lo \" c Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Ro...

    arXiv doi:10.48550/arxiv.2207.04672 2022

Show all 44 references
  1. [9]

    Danilo Croce, Alexandra Zelenanska, and Roberto Basili. 2018. Neural learning for question answering in italian. In AI*IA 2018 -- Advances in Artificial Intelligence, pages 389--402, Cham. Springer International Publishing

    2018

  2. [10]

    Zoltan Csaki, Bo Li, Jonathan Lingjie Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. https://doi.org/10.18653/v1/2024.mrl-1.1 S amba L ingo: Teaching large language models new languages . In Proceedings of the Fourth Wo...

    2024 doi

  3. [11]

    Wietse de Vries and Malvina Nissim. 2021. https://doi.org/10.18653/v1/2021.findings-acl.74 As good as new. how to successfully recycle E nglish GPT -2 to make models for other languages . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836-...

    2021 doi

  4. [12]

    Konstantin Dobler and Gerard de Melo. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.829 FOCUS : Effective embedding initialization for monolingual specialization of multilingual models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    2023 doi

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  7. [15]

    Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. 2022. https://doi.org/10.18653/v1/2022.emnlp-industry.41 Fast vocabulary transfer for language model compression . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: In...

    2022 doi

  8. [16]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.acl-main.740 Don ' t stop pretraining: Adapt language models to domains and tasks . In Proceedings of the 58th Annual Meeting o...

    2020 doi

  9. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  10. [18]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon ...

    2022

  11. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  12. [20]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  13. [21]

    Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.833 I ndo BERT weet: A pretrained language model for I ndonesian T witter with effective domain-specific vocabulary initialization . In Proceedings of the 2021 Conference on Empiric...

    2021 doi

  14. [22]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  15. [23]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \'e , et al. 2023. https://arxiv.org/abs/2211.05100 Bloom: A 176b-parameter open-access multilingual langu...

    2023 arXiv

  16. [24]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  17. [25]

    Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.68 OFA : A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining . In Findings of the Association for C...

    2024 doi

  18. [26]

    Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodol \`a . 2024. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \` e re, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, L \' e onard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro - Ros, Ambrose ...

    2024 doi arXiv

  20. [28]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. https://doi.org/10.18653/v1/2022.naacl-main.293 WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models . In Proceedings of the 2022 Conference of the Nor...

    2022 doi

  21. [29]

    Luca Moroni, Simone Conia, Federico Martelli, and Roberto Navigli. 2024. https://clic2024.ilc.cnr.it/wp-content/uploads/2024/12/66_main_long.pdf ITA-Bench : Towards a more comprehensive evaluation for I talian LLMs . In Proceedings of the Tenth Italian Conference on Computatio...

    2024

  22. [30]

    Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodol \` a . 2023. https://openreview.net/forum?id=SrC-nwieGJ Relative representations enable zero-shot latent space communication . In The Eleventh International Conference on ...

    2023

  23. [31]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. https://aclanthology.org/2024.lrec-main.377 C ultura X : A cleaned, enormous, and multilingual dataset for large language models in 167 langu...

    2024

  24. [32]

    Riccardo Orlando, Luca Moroni, Pere-Llu \' s Huguet Cabot, Edoardo Barba, Simone Conia, Sergio Orlandini, Giuseppe Fiameni, Roberto Navigli, et al. 2024. Minerva llms: The first family of large language models trained from scratch on italian data. In Proceedings of the Tenth I...

    2024

  25. [33]

    Malte Ostendorff and Georg Rehm. 2023. Efficient language model training through cross-lingual and progressive transfer learning. arXiv preprint arXiv:2301.09626

    2023 arXiv

  26. [34]

    Marco Polignano, Pierpaolo Basile, and Giovanni Semeraro. 2024. https://arxiv.org/abs/2405.07101 Advanced natural-based interaction for the italian language: Llamantino-3-anita . Preprint, arXiv:2405.07101

    2024 arXiv

  27. [35]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  28. [36]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  29. [37]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52 COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Pro...

    2022

  30. [38]

    Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul R \"o ttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.findings-acl.441 `` my answer is C '' : First-token probabilities do not match text answers in instruction-tuned langu...

    2024 doi

  31. [39]

    Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher R \' e , Irina Rish, and Ce Zhang....

    2024 doi arXiv

  32. [40]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. https://doi.org/10.18653/v1/W17-4413 Crowdsourcing multiple choice science questions . In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94--106, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  33. [41]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  34. [42]

    Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. https://aclanthology.org/2021.ccl-1.108 A robustly optimized BERT pre-training approach with post-training . In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218--1227, Huhhot, China....

    2021

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.