Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

TokAlign: Efficient Vocabulary Adaptation via Token Alignment

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TokAlign claims that an LLM's vocabulary can be swapped to any target tokenizer by aligning tokens on co-occurrence statistics, cutting initialization perplexity from 340 to 120 and restoring vanilla performance in about 5,000 steps.

desk verdict TokAlign's vocabulary-alignment recipe is genuinely useful and cheap, but its headline distillation boost is confounded and needs a controlled experiment before that claim stands. read the letter →

arxiv 2506.03523 v1 pith:IV4CGDNR submitted 2025-06-04 cs.CL

Chong Li , Jiajun Zhang , Chengqing Zong This is my paper
classification cs.CL
keywords vocabularyadaptationtokenizerreplacementtokenalignmentGloVeembeddingstoken-leveldistillationcross-lingualtransferLLMinitialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are chained to their tokenizers: a tokenizer that fragments new languages or domains slows training and generation, and different vocabularies block token-level knowledge transfer between models. TokAlign tries to break that link by swapping a model's vocabulary to any target tokenizer without retraining from scratch. It aligns every source token to a target token using GloVe embeddings learned from token co-occurrence over 1 billion tokens, then initializes the new embedding matrix and output head from the matched source parameters. The paper reports that this initialization lowers perplexity from 340 with strong baselines to 120 after initialization, and that about 5,000 tuning steps restore vanilla performance. Unifying vocabularies this way also makes token-level distillation roughly 4.4% better than sentence-level distillation on 235 million tokens.

What carries the argument

The load-bearing object is the token-token alignment matrix $M_{s\to t}$, a one-to-one mapping learned as the cosine-similarity argmax between GloVe embeddings of source and target tokens trained on a 1B-token mixed corpus (40% CulturaX, 30% The Stack, 30% Proof-Pile-2). GloVe is used because it encodes global token co-occurrence statistics, which the paper argues survive re-tokenization of the same corpus. The matrix is evaluated by two metrics, BLEU-1 on the recovered token-ID sequence and BERTScore between the de-tokenized recovered text and the original text, and those metrics correlate negatively with initial training loss. The same matrix also drives the progressive two-stage adaptation: embeddings and lm_head are tuned first, then all parameters, which stabilizes convergence.

What would settle it

Run TokAlign on a target language that is absent from the GloVe training corpus and has near-zero vocabulary overlap, and compare its initial perplexity before any tuning against random initialization. If the aligned initialization is not meaningfully better, the co-occurrence-similarity assumption behind the claimed speed-up fails.

Watch

Extended reading notes

Core claim

The central claim is that token IDs from two different tokenizers can be aligned one-to-one purely from statistical co-occurrence, and that this alignment provides a reusable initialization for vocabulary replacement. Specifically, TokAlign trains GloVe embeddings on the same corpus tokenized with both tokenizers, computes pairwise cosine similarity, and records a one-to-one mapping matrix $M_{s\to t}$; shared tokens are replaced directly. Target vocabulary embeddings and the language-modeling head are initialized from the most similar source token, followed by a progressive two-stage fine-tune, first on vocabulary-related parameters and then on all parameters. The paper asserts this brings the first-step training loss down from 17.8 to 9.5 on Pythia 2.8B, preserves 97.6% of vanilla performance after 5,000 steps even when the target Gemma vocabulary shares only 6.23% overlap with Pythia, and yields a 1B model comparable to a vanilla 7B model after token-level distillation.

Load-bearing premise

The load-bearing premise is that cosine similarity among GloVe token embeddings trained on 1B tokens of a mixed corpus assigns each new target token to a source token close enough to make the model recover quickly, even when no semantic relationship is visible.

Editorial extensions

If this is right

  • A model can be retargeted to a multilingual tokenizer for cross-lingual transfer, shrinking tokenized text length by 29.2% on average across 13 languages while largely preserving English ability.
  • Vocabulary replacement becomes cheap enough that matching a small model's vocabulary to a stronger teacher's vocabulary is practical, enabling token-level distillation that improves the base model by 4.4% over sentence-level distillation on 235M tokens.
  • Recovery time scales with vocabulary overlap: with 32.85% overlap, 5,000 steps recover about 99.1% of vanilla performance; with 6.23% overlap, about 97.6%.
  • The two-stage tuning procedure prevents loss spikes at high learning rates, allowing more aggressive adaptation.
  • The results generalize across Pythia 1B, 2.8B, and 6.9B and across Gemma, Qwen2, LLaMA2, and LLaMA3 target tokenizers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is to use BLEU-1 and BERTScore as a cheap pre-screening: compute alignment quality on a small sample before committing fine-tuning compute, since the reported negative correlation with initial loss suggests alignment quality predicts benefit.
  • For closed-weights models whose pretraining corpus is unknown, the method's portability depends on whether a corpus can be reconstructed from the BPE vocabulary; if that reconstruction fails, TokAlign would be confined to open models.
  • Because the qualitative alignments are often co-occurring rather than synonymous, the method may work better for transfer that depends on contextual statistics than for tasks requiring exact lexical semantic equivalence; a testbed separating those task types would clarify the boundary.
  • The same co-occurrence alignment logic could be applied to non-textual tokens, such as multimodal tokens, which the paper names as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes TokAlign, a method for replacing the vocabulary of a pretrained LLM by aligning source-vocabulary tokens to target-vocabulary tokens using GloVe embeddings trained on a 1B-token mixed corpus. The learned one-to-one mapping initializes the new embedding matrix and language-modeling head, followed by a two-stage fine-tuning procedure that first tunes only vocabulary-related parameters and then the full model. The authors report large reductions in initial perplexity (from 3.4e2 for ZeTT to 1.2e2), improved multilingual compression rates, recovery of vanilla performance in roughly 5k steps, and a +4.4% improvement of token-level distillation over sentence-level distillation after unifying vocabularies. Experiments cover Pythia models of three sizes and four target tokenizers, with robustness checks using SlimPajama and relative representations.

Significance. If the claims hold, TokAlign would be a practical, low-cost alternative to hypernetwork-based vocabulary adaptation, offering a simple CPU-only alignment step plus a modest fine-tuning budget. The paper is broad in scope: it evaluates cross-lingual transfer on four multilingual benchmarks, vocabulary replacement on six English benchmarks, and distillation from three teacher models, across three model scales. The availability of code, the SlimPajama robustness check, and the relative-representation variant are concrete strengths. However, the two load-bearing empirical claims—the +4.4% distillation benefit and the 5% significance of the vocabulary-adaptation gains—are not cleanly supported as reported, and the qualitative case study undercuts the semantic-alignment framing of the method.

major comments (3)
  1. [§4.2.2, Table 3] The +4.4% improvement of token-level distillation over sentence-level distillation is confounded. The "+ Sentence distill" baseline retains the original Pythia vocabulary and trains only on teacher-generated text, whereas the token-level distillation condition first replaces the Pythia vocabulary with the Qwen2 vocabulary via TokAlign and then trains on KL divergence against teacher logits plus cross-entropy on gold tokens. These conditions differ in both the vocabulary (unified vs. original) and the training signal (logits plus gold labels vs. sampled text). The reported gain therefore cannot be attributed to vocabulary unification, contrary to the abstract's claim that "After unifying vocabularies between LLMs, token-level distillation can remarkably boost." A controlled comparison holding the vocabulary fixed across both distillation objectives is needed before this central claim can be accepted.
  2. [Table 4] Table 4 marks TokAlign results with asterisks and the caption states "* indicates statistically significant improvements of 5% level," but the paper does not describe the significance test, the number of seeds, the variance, or the comparison baseline. Without this information the significance claim is unverifiable. Please specify the test (e.g., paired bootstrap, seed-based confidence intervals) and report the underlying uncertainty, or remove the asterisks.
  3. [Appendix B.5, Table 9] The qualitative case study reports that "in most cases, a clear semantic relationship between two aligned tokens cannot be found" and offers co-occurrence rather than semantic similarity as the explanation. This is in tension with the paper's framing that token-token co-occurrence preserves "semantic and syntactic information" (Section 1) and with the use of cosine similarity of GloVe embeddings as the alignment criterion. The aggregate BLEU-1/BERTScore correlations in Figure 5 are suggestive but do not validate the semantic content of the mapping. The paper should either provide a mechanism-level validation (e.g., ablations showing that the alignment quality, not just the initialization statistics, drives the gains) or substantially soften the semantic-alignment language throughout the abstract and introduction.
minor comments (7)
  1. [Appendix A] The hyper-parameter description says "The max training iteration and the size of the slide window are 15," which is ambiguous: are both the number of iterations and the window size 15? Please state the values separately.
  2. [Appendix A / §4.2.2] The learning rate used for TokAlign in the main experiments is not reported; only the baseline learning rate (5e-5) is given. Please report the learning rate (and schedule) used for the TokAlign rows in Tables 1, 3, and 4.
  3. [§4.2.2] The sentence "The proportion of training samples is empirically set to 15%" is unclear—15% of which mixture? Please specify the ratio of downstream-task samples to Pile samples and how the 15% figure is applied.
  4. [Figure 4 / §4.4] The claim that TokAlign is "1.92x faster" is based on the number of training steps to reach a given loss (2.5k vs. 1.3k), not wall-clock time. This should be stated explicitly to avoid conflating step count with computational cost.
  5. [Abstract and §4.3] The abstract says "costs as few as 5k steps to restore the performance of the vanilla model," but Table 5 and Section 4.3 report recovery of 97.6–99.1%, not 100%. Please use phrasing such as "nearly restore" or "recover most of the performance."
  6. [Appendix B.5] There is a typo: "it is better to choose a matric to quantify" should read "a metric."
  7. [§4.1 / Table 1] The perplexity in Table 1 is normalized to the Pythia vocabulary following Wei et al. (2023). Since the normalization method is important for interpreting the absolute values, please provide a one-sentence description of the normalization or a pointer to the exact equation in the cited work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central initialization and distillation claims are supported by external benchmarks and robustness checks.

full rationale

Walking the claimed derivation chain, the core TokAlign pipeline (GloVe token-representation learning on a mixed corpus, cosine-similarity alignment, parameter rearrangement, two-stage tuning) is self-contained: the alignment matrix is trained without using downstream evaluation labels, and the reported gains are measured on held-out perplexity and external downstream tasks (ARC, BoolQ, HellaSwag, OpenbookQA, PIQA, WinoGrande, XNLI, PAWS-X, XCOPA, XStoryCloze) rather than on quantities defined by the method. The robustness check replacing the alignment corpus with SlimPajama (Table 4, 'w/ SlimPajama') provides independent grounding against the concern that the GloVe embeddings were tuned to the CulturaX evaluation distribution. The Limitations section explicitly acknowledges the pre-training-corpus-availability assumption, and the Appendix B.5 case study admits that aligned tokens often lack semantic relationship; these are honest scope statements, not circular reasoning. The only caveats are experimental-design issues, not circularity: the Section 4.2.2 comparison of token-level distillation (which uses the unified Qwen2 vocabulary plus KL and gold-token CE) against '+ Sentence distill' (which keeps the original Pythia vocabulary and uses only teacher-generated text) varies two factors at once, so the +4.4% cannot be cleanly attributed to vocabulary unification alone; and the compression-rate improvement in Table 1 is largely inherited from the chosen target tokenizer rather than from the alignment. Neither caveat makes a result equivalent to its inputs by construction. Self-citations (Li et al. 2024; Yang et al. 2023; Xu et al. 2024) appear only in related-work context and are not load-bearing. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. Score 0.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The method relies on the distributional hypothesis applied to BPE tokens, availability of a proxy pre-training corpus, and sufficient vocabulary coverage from 1B GloVe tokens. It introduces no new physical or conceptual entities. The main hand-chosen values are the GloVe hyperparameters, corpus mix, token budget, adaptation step count, two-stage split, and the 15% distillation sample proportion.

free parameters (9)
  • GloVe embedding dimension = 300
    Chosen empirically in Appendix A; standard for GloVe, not ablated.
  • GloVe window size = 15
    Set to 15 in Appendix A; not ablated.
  • GloVe training iterations = 15
    Set to 15 in Appendix A; not ablated.
  • Corpus mixture ratios = CulturaX 40%, The Stack 30%, Proof-Pile-2 30%
    Empirically chosen composition for GloVe training corpus in Section 3.1 Step 1; not systematically varied.
  • GloVe training token budget = 1B tokens
    Investigated in Appendix B.2; more than 1B not tested due to compute budget.
  • Distillation training sample proportion = 15%
    Empirically set to 15% in Section 4.2.2 to avoid degradation in language modeling performance.
  • Two-stage tuning split = 50%/50%
    Stage 1 tunes embedding and lm_head for the first half of steps, then full tuning, per Section 3.3.
  • Number of adaptation steps = 5k steps (10B tokens)
    Default budget for vocabulary adaptation; Appendix B.4 explores a 2k-step budget.
  • Learning rates = 5e-5 for baselines, up to 6.4e-4 for TokAlign two-stage
    Chosen to reduce loss spikes (Appendix A, Figure 6); not a fitted constant but affects convergence.
assumptions (4)
  • domain assumption Distributional hypothesis: token co-occurrence statistics capture transferable semantic information across tokenizers.
    Foundation of GloVe-based alignment in Section 3.1 Steps 2-3; the method assumes cosine similarity in this space is meaningful for token matching.
  • domain assumption Pre-training corpus distribution of the base model is available or inferable.
    Stated in the Limitations section; needed to choose the GloVe training corpus. The authors use Pile for fine-tuning and a mixed corpus for GloVe.
  • domain assumption GloVe embeddings trained on 1B tokens cover most of the target vocabularies (>90%).
    Empirically verified in Appendix B.2, but coverage is a precondition for alignment quality; low coverage would leave many target tokens without a good source match.
  • standard math Cosine similarity of token embeddings is a valid similarity measure for token alignment.
    Used in Section 3.1 Step 3 for pair-wise similarity; standard vector-space assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TokAlign: Efficient Vocabulary Adaptation via Token Alignment." pith.science (2026). https://pith.science/paper/IV4CGDNR

@misc{pith2026250603523,
  author       = {Pith},
  title        = {Pith review of: TokAlign: Efficient Vocabulary Adaptation via Token Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IV4CGDNR}},
  note         = {Machine review of arXiv:2506.03523}
}
abstract

Tokenization serves as a foundational step for Large Language Models (LLMs) to process text. In new domains or languages, the inefficiency of the tokenizer will slow down the training and generation of LLM. The mismatch in vocabulary also hinders deep knowledge transfer between LLMs like token-level distillation. To mitigate this gap, we propose an efficient method named TokAlign to replace the vocabulary of LLM from the token co-occurrences view, and further transfer the token-level knowledge between models. It first aligns the source vocabulary to the target one by learning a one-to-one mapping matrix for token IDs. Model parameters, including embeddings, are rearranged and progressively fine-tuned for the new vocabulary. Our method significantly improves multilingual text compression rates and vocabulary initialization for LLMs, decreasing the perplexity from 3.4$\text{e}^2$ of strong baseline methods to 1.2$\text{e}^2$ after initialization. Experimental results on models across multiple parameter scales demonstrate the effectiveness and generalization of TokAlign, which costs as few as 5k steps to restore the performance of the vanilla model. After unifying vocabularies between LLMs, token-level distillation can remarkably boost (+4.4% than sentence-level distillation) the base model, costing only 235M tokens.

Figures

Figures reproduced from arXiv: 2506.03523 by the authors.

Figure 1
Figure 1. The compression rates of tokenizers across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of TokAlign to align token IDs from different vocabularies. We train token representations on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) We choose BLEU-1 and BERTScore to evaluate the performance of alignment matrix [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The training loss of Pythia [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The relationship between initial training loss [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The loss curve of Pythia1B under two-stage tuning or direct full parameters tuning. 0.5 1.0 1.5 2.0 2.5 Steps (k) 0 2 4 6 8 10 Loss 8e 5 1.6e 4 3.2e 4 6.4e 4 1.28e 3 2.56e 3 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The training loss curve of Pythia1B for learn￾ing rate used during replacing to the Gemma tokenizer. 5 Conclusion and Future Work In this paper, we introduce a method named TokAlign to replace the tokenizer of large language models from a token-token co-occurrence view…
Figure 8
Figure 8. Figure 8: The average vocabulary coverage (a) and initial training loss of Pythia [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The training loss curve of Pythia1B for differ￾ent overlapping ratios. 0.5 1.0 1.5 2.0 2.5 Steps (k) 0 2 4 6 8 10 12 14 Loss Gemma (6.23%) Qwen2 (26.92%) LLaMA3 (32.85%) [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The training loss to different tokenizers using [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. In-Place Tokenizer Expansion for Pre-trained LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Continuing a model's own BPE merges and training only new embedding rows preserves quality while cutting token counts 2.4–4× for previously under-tokenized languages.

  2. The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Reasoning messages between heterogeneous VLMs can be routed through the image-token span: a distilled universal codec plus affine alignment transmits latent traces across model families, cutting wall-clock time in sma...

Reference graph

Works this paper leans on

78 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2024. https://openreview.net/forum?id=4WnqRR915j Llemma: An open language model for mathematics . In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Gunjan Balde, Soumyadeep Roy, Mainack Mondal, and Niloy Ganguly. 2024. https://doi.org/10.24963/ijcai.2024/683 Medvoc: vocabulary adaptation for fine-tuning pre-trained language models on medical text summarization . In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI '24

    doi:10.24963/ijcai.2024/683 2024

  3. [3]

    Yoshua Bengio, R \'e jean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137--1155

    work page 2003

  4. [4]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. https://proceedings.mlr.press/v202/biderman23a.html Pythia: A suite for analyzing large language models across trai...

    2023

  5. [5]

    Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. https://doi.org/10.1609/aaai.v34i05.6239 Piqa: Reasoning about physical commonsense in natural language . In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The...

    doi:10.1609/aaai.v34i05.6239 2020

  6. [6]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. https://arxiv.org/abs/1607.04606 Enriching word vectors with subword information . arXiv preprint arXiv:1607.04606

    arXiv 2017

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  8. [8]

    Chau, Lucy H

    Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.118 Parsing with multilingual BERT , a small corpus, and a small treebank . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324--1334, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.118 2020

Show all 78 references
  1. [9]

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555

    2014 arXiv

  2. [10]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  3. [11]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    2018 arXiv

  4. [12]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods ...

    2018 doi

  5. [13]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R\' e . 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf Flashattention: Fast and memory-efficient exact attention with io-awareness . In Advances ...

    2022

  6. [14]

    Konstantin Dobler and Gerard de Melo. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.829 FOCUS : Effective embedding initialization for monolingual specialization of multilingual models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    2023 doi

  7. [15]

    Downey, Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeld

    C.m. Downey, Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeld. 2023. https://doi.org/10.18653/v1/2023.mrl-1.20 Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages . In Proceedings of the 3rd Workshop on Multi-lingual Rep...

    2023 doi

  8. [16]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . P...

    2020 arXiv

  9. [17]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  10. [18]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, and et al. 2024. https://arxiv.org/abs/2406.12793 Chatglm: A family of large language models from glm-130b to glm-4 all tools . Preprint, arXiv:2406.12793

    2024 arXiv

  11. [19]

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. 2024. Data mixture inference: What do bpe tokenizers reveal about their training data? arXiv preprint arXiv:2407.16607

    2024 arXiv

  12. [20]

    S Hochreiter. 1997. Long short-term memory. Neural Computation MIT-Press

    1997

  13. [21]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  14. [22]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  15. [23]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  16. [24]

    Yoon Kim and Alexander M. Rush. 2016. https://doi.org/10.18653/v1/D16-1139 Sequence-level knowledge distillation . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317--1327, Austin, Texas. Association for Computational Linguistics

    2016 doi

  17. [25]

    Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Mu \ n oz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. 2023. https://openreview.net/forum?id=pxpbTdUEpD The stack: 3 TB...

    2023

  18. [26]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018 doi

  19. [27]

    Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.28 Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback . In Pro...

    2023 doi

  20. [28]

    Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.445 Improving in-context learning of multilingual generative language models with cross-lingual alignment . In Proceedings of the 2024 Conference of the North American C...

    2024 doi

  21. [29]

    Haoran Li, Jiajun Zhang, and Chengqing Zong. 2017. Implicit discourse relation recognition for english and chinese with multiview modeling and effective representation learning. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(3):1--21

    2017

  22. [30]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022 doi

  23. [31]

    Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.68 OFA : A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining . In Findings of the Association for C...

    2024 doi

  24. [32]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  25. [33]

    Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. 2024. https://arxiv.org/abs/2407.06089 Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models . Preprint, arXiv:2407.06089

    2024 arXiv

  26. [34]

    Meta. 2024. https://ai.meta.com/blog/meta-llama-3/ Introducing meta llama 3: The most capable openly available llm to date . Qwen blog

    2024

  27. [35]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed precision training. In International Conference on Learning Representations

    2018

  28. [36]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  29. [37]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 a . Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

    2013 arXiv

  30. [38]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 b . Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546

    2013 arXiv

  31. [39]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. https://doi.org/10.18653/v1/2022.naacl-main.293 WECHSEL : Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models . In Proceedings of the 2022 Conference of the Nor...

    2022 doi

  32. [40]

    Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. 2024. https://arxiv.org/abs/2405.07883 Zero-shot tokenizer transfer . Preprint, arXiv:2405.07883

    2024

  33. [41]

    Edoardo Mosca, Mohamed Hesham Ibrahim Abdalla, Paolo Basso, Margherita Musumeci, and Georg Groh. 2023. https://doi.org/10.18653/v1/2023.trustnlp-1.17 Distinguishing fact from fiction: A benchmark dataset for identifying machine-generated scientific papers in the LLM era. In Pr...

    2023 doi

  34. [42]

    Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodol \`a . 2023. https://openreview.net/forum?id=SrC-nwieGJ Relative representations enable zero-shot latent space communication . In The Eleventh International Conference on L...

    2023

  35. [43]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward ...

    2023 doi

  36. [44]

    Nandini Mundra, Aditya Nanda Kishore Khandavally, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M Khapra. 2024. https://aclanthology.org/2024.conll-1.8 An empirical comparison of vocabulary expansion and initialization approaches for language models . In Proceed...

    2024

  37. [45]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. https://aclanthology.org/2024.lrec-main.377 C ultura X : A cleaned, enormous, and multilingual dataset for large language models in 167 langu...

    2024

  38. [46]

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. arXiv preprint arXiv:2309.09400

    2023 arXiv

  39. [47]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    2023 arXiv

  40. [48]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G lo V e: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doh...

    2014 doi

  41. [49]

    Edoardo Maria Ponti, Goran Glava s , Olga Majewska, Qianchu Liu, Ivan Vuli \'c , and Anna Korhonen. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.185 XCOPA : A multilingual dataset for causal commonsense reasoning . In Proceedings of the 2020 Conference on Empirical Method...

    2020 doi

  42. [50]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://openai.com/blog/language-unsupervised/ Improving language understanding by generative pre-training . OpenAI blog

    2018

  43. [51]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. https://openai.com/blog/better-language-models/ Language models are unsupervised multitask learners . OpenAI blog

    2019

  44. [52]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. https://doi.org/10.1145/3394486.3406703 Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters . In Proceedings of the 26th ACM SIGKDD International Conferenc...

    2020

  45. [53]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. https://doi.org/10.1609/aaai.v34i05.6399 Winogrande: An adversarial winograd schema challenge at scale . Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732--8740

    2020 doi

  46. [54]

    BigScience Workshop: Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, and Alexandra Sasha Luccioni et al. 2023. https://arxiv.org/abs/2211.05100 Bloom: A 176b-parameter open-access multilingual language model . arXiv pre...

    2023 arXiv

  47. [55]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  48. [56]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    2023

  49. [57]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/c3a690be93aa602ee2dc0ccab5b7b67e-Paper.pdf Mpnet: Masked and permuted pre-training for language understanding . In Advances in Neural Information Proces...

    2020

  50. [58]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  51. [59]

    Alexey Tikhonov and Max Ryabinin. 2021. https://doi.org/10.18653/v1/2021.findings-acl.310 I t ' s A ll in the H eads: U sing A ttention H eads as a B aseline for C ross- L ingual T ransfer in C ommonsense R easoning . In Findings of the Association for Computational Linguistic...

    2021 doi

  52. [60]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://arxiv.org/abs/2302.13971 Lla...

    2023 arXiv

  53. [61]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  54. [62]

    Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306

    2020 arXiv

  55. [63]

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D ' souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. ht...

    2024 doi

  56. [64]

    Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2018. https://doi.org/10.1609/AAAI.V32I1.12031 Learning multimodal word representation via dynamic fusion methods . In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Ap...

    2018 doi

  57. [65]

    Jingxuan Wei, Linzhuang Sun, Yichong Leng, Xu Tan, Bihui Yu, and Ruifeng Guo. 2024. https://www.ijcai.org/proceedings/2024/722 Sentence-level or token-level? A comprehensive study on knowledge distillation . In Proceedings of the Thirty-Third International Joint Conference on ...

    2024

  58. [66]

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei L \"u , Rui Hu, et al. 2023. Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341

    2023 arXiv

  59. [67]

    T Wolf. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  60. [68]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith St...

    2016 arXiv

  61. [69]

    Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.395 Bridging the gap between different vocabularies for LLM ensemble . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui...

    2024 doi

  62. [70]

    Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024. https://aclanthology.org/2024.findings-emnlp.396 An empirical study on cross-lingual vocabulary adaptation for efficient language model inference . In Findings of the Association for Computational Linguistics: ...

    2024

  63. [71]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  64. [72]

    Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098

    2023 arXiv

  65. [73]

    Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. https://doi.org/10.18653/v1/D19-1382 PAWS - X : A cross-lingual adversarial dataset for paraphrase identification . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the ...

    2019 doi

  66. [74]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  67. [75]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. https://arxiv.org/abs/2401.02385 Tinyllama: An open-source small language model . arXiv preprint arXiv:2401.02385

    2024 arXiv

  68. [76]

    Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.04948

    2023 arXiv

  69. [77]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  70. [78]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.