Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free method transplants tokenizers between pretrained LLMs by reconstructing unseen token embeddings with Orthogonal Matching Pursuit.

desk verdict A sound low-overlap result and a genuinely useful OMP trick are oversold by a high-overlap experiment that shows OMP is statistically indistinguishable from zero and mean initialization. read the letter →

arxiv 2506.06607 v1 pith:NY6O7F6I submitted 2025-06-07 cs.CL cs.AIcs.LG

Charles Goddard , Fernando Fernandes Neto This is my paper
classification cs.CLcs.AIcs.LG
keywords tokenizertransplantationorthogonalmatchingpursuitembeddinginitializationzero-shottransferlargelanguagemodelssparseapproximationnumericaltokenizationvocabularyalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a pretrained model's tokenizer can be swapped for another model's tokenizer with no gradient updates, preserving most of the original model's performance, by reconstructing each unseen token's embedding as a sparse linear combination of shared anchor tokens. The reconstruction is done in the donor model's embedding space, and the same coefficients are then placed into the base model's embedding space. On Llama-to-Mistral-NeMo (12B) and Qwen-to-Llama (1B) transplants, the paper reports that this beats zero-initialization, mean-initialization, and published zero-shot initialization methods across MMLU, ARC, XNLI, GSM8K, LAMBADA, and perplexity benchmarks. The paper identifies mismatched numerical tokenization schemes (single-digit versus triplet chunking) as the main cause of large drops in mathematical reasoning, and supports this by showing that a transplant between models with matched numeric schemes loses only about five percent on GSM8K. If the method holds up, tokenizer replacement becomes a cheap post hoc operation, enabling cross-tokenizer distillation, speculative decoding, ensembling, and domain vocabulary expansion.

What carries the argument

The central object is Orthogonal Matching Pursuit (OMP), a greedy sparse-recovery algorithm that iteratively selects the shared-token embedding most correlated with the current residual, solves a least-squares fit on the chosen anchors, and updates the residual. The paper's trick is to run OMP entirely in the donor embedding space and transplant only the signed coefficients into the base embedding space, so the two spaces' dimensionalities never need to match. The geometric justification is approximate orthogonal equivalence: if donor and base embedding spaces are related by a near-orthogonal transformation on the shared vocabulary, then applying that transformation to a sparse donor-space combination yields approximately the same combination in the base space. An incremental QR decomposition of the selected anchor columns keeps per-iteration cost linear, making the transfer practical for vocabularies of roughly one hundred thousand tokens.

What would settle it

Take one of the paper's tokenizer pairs and compute a Procrustes orthogonal map between the shared-token embeddings; if the residual on unseen but semantically known tokens is not substantially smaller than the residual of a random orthogonal map, the alignment premise fails. On a more direct level, transplant a new pair of LLMs with mismatched numeric schemes using OMP-K64 and compare its benchmark scores with mean-initialization; a reversal of the reported ordering would refute the claim that OMP best preserves zero-shot performance.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that Orthogonal Matching Pursuit gives a training-free way to construct a new vocabulary embedding matrix that is consistent with the base model's embedding geometry, and that this preservation is consistently better across classification, reasoning, and perplexity benchmarks than zero-initialization, mean-initialization, and the existing zero-shot initialization methods tested. The central operation is to approximate each out-of-vocabulary token as a sparse signed combination of shared anchor embeddings in the donor space, then transplant exactly those coefficients into the base space. The paper further claims that most remaining degradation in mathematical reasoning comes from structural mismatches in numerical tokenization rather than from the transplantation mechanism itself, supported by a control experiment with matched single-digit numeric schemes in which GSM8K drops by about 5.6 percent instead of more than seventy percent.

Load-bearing premise

The method assumes the donor and base embedding spaces are aligned by an approximately orthogonal transformation on the shared vocabulary; if that alignment does not hold, the transplanted coefficients are not guaranteed to represent the same meaning in the base space.

Editorial extensions

If this is right

  • Language models can be retokenized in minutes on a single GPU, enabling the direct reuse of pretrained weights under a new tokenizer without continued pretraining.
  • Teacher and student models with different vocabularies become directly comparable for logit-level distillation, and arbitrary model pairs become usable in speculative decoding without prior vocabulary alignment.
  • OMP can initialize embeddings for new domain-specific tokens by reconstructing them from existing anchors, adding minimal vocabulary overhead while preserving existing model performance.
  • When numeric tokenization schemes are matched, mathematical reasoning can survive transplantation nearly intact, as the paper's GSM8K control experiment shows, so tokenizer-pair selection is itself a practical design lever.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same sparse-coefficient transfer to attention output or feed-forward projection spaces, though those spaces are less isotropic and may not enjoy the same approximate orthogonal alignment.
  • The numerical-tokenization finding predicts a testable rule: OMP transplants between tokenizers that both use single-digit tokens or both use triplet chunking should show much smaller math-reasoning drops than mixed-scheme pairs; a broad survey across tokenizer pairs would establish how general the rule is.
  • The use of signed coefficients lets OMP explore a larger linear subspace than convex-hull methods, which may explain its advantage, but it also means anchors with large opposing coefficients deserve scrutiny as potential sources of instability.
  • A hybrid strategy that applies OMP for general vocabulary while using dedicated initialization for numeric tokens is an explicit, plausible route to recovering most lost mathematical ability, and the paper's control experiment suggests the expected gain is large.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a training-free tokenizer-transplantation method: for each token in the donor vocabulary absent from the base vocabulary, it computes a k-sparse representation of the donor embedding in terms of shared-token donor embeddings using Orthogonal Matching Pursuit, then writes the same coefficients as a linear combination of the corresponding base embeddings. Shared tokens are copied. Experiments transplant Llama 3 into Mistral NeMo 12B and Qwen 2.5 into Llama 3 1B, plus a matched-numeric-scheme control (Mistral NeMo→Qwen 7B), and compare against zero, mean, WECHSEL, FOCUS, ZETT, and CLPTransfer on MMLU, ARC-C, GSM8K, XNLI, AGIEval, LAMBADA, Belebele, and WikiText perplexity. The paper also reports continued-pretraining results and integrates the method into the open-source mergekit-tokensurgeon tool.

Significance. If the empirical claims held uniformly, the method would be a practical, low-cost tool for cross-tokenizer distillation, speculative decoding, and vocabulary expansion, and the paper's release of an efficient incremental-QR OMP implementation plus reproducible harness evaluations would make it directly usable. The Mistral NeMo→Qwen control is a useful falsifiable test of the numerical-tokenization hypothesis, and the finding that matched digit-tokenization schemes reduce GSM8K degradation to -5.6% is a concrete, non-obvious result. The main weakness is that the headline comparative claim is only clearly supported in the low-overlap Llama→Mistral NeMo experiment; in the high-overlap Qwen→Llama experiment the OMP and trivial baselines are indistinguishable on several primary metrics. This does not void the contribution but requires scoping or additional statistical support before the abstract's stronger wording is justified.

major comments (3)
  1. [Abstract; Section 1; Table 3; Table 1] The claim that OMP preserves performance 'far better than simple heuristics (zero, mean)' is contradicted by the Qwen→Llama results. In Table 3, OMP-K64, ZeroEmbed, and MeanEmbed produce identical values to four decimals on MMLU (0.3640/0.3640/0.3641), ARC-C (0.3626/0.3626/0.3626), and GSM8K (0.0144/0.0144/0.0144); Table 1 reports bootstrap standard errors of 0.004, 0.014, and 0.013 for these benchmarks, so the observed differences are within noise. Section 4.2's statement that 'This performance outperforms all simpler heuristics' is not supported by these numbers. Please either restrict the comparative claim to low-overlap settings, add significance tests or confidence intervals that demonstrate separation, or explicitly present the high-overlap case as a case where OMP is competitive but not superior.
  2. [Section 3.3] The transfer of sparse coefficients from donor to base space is justified by 'Approximate Orthogonal Equivalence' plus sparse-reconstruction stability, and the paper explicitly disclaims proving such an alignment. Because this is the load-bearing geometric premise, the manuscript should provide direct empirical evidence: for example, estimate an orthogonal map U on shared tokens and report the alignment error ‖U e^donor_j − e^base_j‖, or compare the OMP reconstruction residual in donor space with the actual error of the transplanted embedding in base space. Without such a check, the reader cannot distinguish a method that works because of the stated geometry from one that works for other reasons (e.g., the shared tokens dominating benchmark inputs).
  3. [Section 4.3] The sparsity level k=64 is selected after scanning the same benchmarks (MMLU, ARC, GSM8K, etc.) used in the main comparison tables, and only the selected value is used in the headline comparisons (Tables 2–4). This makes the reported OMP numbers post-selection and partly optimistic relative to the fixed baselines. Please report results for all k values in the main comparison or choose k on a validation split, and state whether the conclusions change for k=8 or k=16. The effect is likely modest, but it affects the claim that OMP 'consistently achieves the best overall performance'.
minor comments (6)
  1. [Abstract] The abstract contains 'atraining-free' with a missing space.
  2. [Section 1] The sentence begins 'Inspeculative decoding' with a missing space before 'speculative'.
  3. [Table 3] The CLPTransfer WikiText entry of 6791.6044 bits/byte and the corresponding '+ 1e6%' relative change are awkwardly formatted; use scientific notation and consistent rounding.
  4. [Appendix A.1] The sparse decomposition display is a table but is captioned as 'Figure 1'; renumber or recaption it for consistency.
  5. [Table 2] The FOCUS row reports LAMBADA accuracy 0.0000; if this is literally zero correct predictions, state that explicitly, since such an exact zero is surprising and may indicate a degenerate output.
  6. [Section 4.5] The sentence 'zero-shot OMP alone already outperforms other zero-shot heuristics' is too broad: in Table 5, OMP-K64 zero-shot XNLI (0.3430) is below ZETT (0.3437) and FOCUS (0.3578). Qualify the sentence to say 'overall' or 'on most benchmarks'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: OMP embeddings are constructed from donor-space sparse codes and tested on external benchmarks; the unproven alignment assumption and k-selection are limitations, not circular reductions.

full rationale

The derivation chain is: (1) compute sparse coefficients for each unseen token from donor embeddings of shared anchors; (2) transplant those same coefficients into the base embedding space; (3) evaluate the resulting model on external benchmarks. No step fits a parameter to benchmark targets, and the benchmark results are not used to construct the transplanted embeddings. The geometric justification in Section 3.3 explicitly relies on an assumed approximate orthogonal equivalence and states 'we do not prove (or seek to prove) this alignment universally exists'; this is an unverified premise that limits the method's guarantee, but it is not circular because the empirical results do not feed back into the construction. The only self-citation is the authors' own mergekit [13], used for software context and application scenarios, not to justify the central method or its evaluation, so it is not load-bearing. The sparsity level k=64 is recommended after inspecting the same benchmark tables, which is selection on the evaluation set and weakens the strength of the 'best overall' claim, but all k values are reported and the choice is a model hyperparameter, not a fitted quantity that forces the predicted outcome. In the high-overlap Qwen→Llama experiment, OMP, ZeroEmbed, and MeanEmbed are statistically tied on MMLU, ARC, and GSM8K, which contradicts the abstract's 'far better' phrasing; this is an overstatement and an experimental confound (most benchmark tokens are shared), not a circular derivation. Overall, the paper's core result is self-contained and externally evaluated; the flagged issues are correctness and assumption-strength concerns, not circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method rests on two unproved structural assumptions about embedding geometry. First, the donor and base embedding spaces are related by an approximately orthogonal transformation on the shared-vocabulary subspace, so that linear coefficients computed in one space remain meaningful in the other. Second, the shared-token dictionary is coherent enough for OMP to recover sparse codes that represent donor tokens well. Both are asserted from prior literature and from the paper's own experiments, not derived. No new entities are posited, and the only free parameter is the sparsity level k.

free parameters (1)
  • sparsity level k = 64
    Number of OMP atoms chosen by sweeping k in {8,16,32,64,256} on the evaluation benchmarks (Section 4.3); the recommended value 64 is used for the headline results.
assumptions (3)
  • domain assumption Approximate orthogonal equivalence: there exists U with U U^T ≈ I and U^T U ≈ I such that U e_donor_j ≈ e_base_j for shared tokens j.
    Stated in Section 3.3 as 'Approximate Orthogonal Equivalence'; the paper explicitly says it does not prove this alignment universally exists.
  • domain assumption OMP recovery: the dictionary of shared anchor embeddings satisfies incoherence or RIP sufficient for OMP to approximate donor tokens with small error.
    Invoked in Section 3.3 via Sparse Reconstruction Stability; no verification that the actual embedding dictionary meets these conditions.
  • domain assumption Shared-token semantic alignment: overlapping tokens have comparable meanings in both models so that copying or combining their embeddings preserves semantics.
    Used throughout the method (Section 3.1, Case 1) and implicit in the coefficient transfer; if shared tokens are not semantically aligned, the base-space combination is meaningless.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit." pith.science (2026). https://pith.science/paper/NY6O7F6I

@misc{pith2026250606607,
  author       = {Pith},
  title        = {Pith review of: Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NY6O7F6I}},
  note         = {Machine review of arXiv:2506.06607}
}
abstract

We present a training-free method to transplant tokenizers in pretrained large language models (LLMs) by reconstructing unseen token embeddings via Orthogonal Matching Pursuit (OMP). Specifically, we approximate each out-of-vocabulary token as a sparse linear combination of shared tokens, in two phases: first, compute each new token's representation in the donor embedding space with a small dictionary of shared anchor tokens, then transfer these same sparse coefficients back into the base model's embedding space. On two challenging cross-tokenizer tasks--Llama$\to$Mistral NeMo (12B) and Qwen$\to$Llama (1B)--we show that OMP achieves best zero-shot preservation of the base model's performance across multiple benchmarks, while other zero-shot approaches degrade significantly. Compared to baselines (zero-init, mean-init, and existing approaches like WECHSEL, FOCUS, ZETT), OMP consistently achieves the best overall performance, effectively bridging large tokenizer discrepancies without gradient updates. Our analysis further identifies mismatched numerical tokenization schemes as a critical challenge for preserving mathematical reasoning capabilities. This technique enables direct reuse of pretrained model weights with new tokenizers, facilitating cross-tokenizer knowledge distillation, speculative decoding, ensembling, merging, and domain-specific vocabulary adaptations. We integrate our method into the open-source mergekit-tokensurgeon tool for post hoc vocabulary realignment.

Figures

Figures reproduced from arXiv: 2506.06607 by the authors.

Figure 1
Figure 1. Sparse linear decompositions of selected tokens from Qwen 2.5’s vocabulary. Each token [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. In-Place Tokenizer Expansion for Pre-trained LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Continuing a model's own BPE merges and training only new embedding rows preserves quality while cutting token counts 2.4–4× for previously under-tokenized languages.

  2. The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Reasoning messages between heterogeneous VLMs can be routed through the image-token span: a distilled universal codec plus affine alignment transmits latent traces across model families, cutting wall-clock time in sma...

Reference graph

Works this paper leans on

36 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    Hydra: Sequentially-dependent draft heads for medusa decoding

    Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109 , 2024

    arXiv 2024

  2. [2]

    On the cross-lingual transferability of monolingual representations

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors,Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online, July 2020. Association for Computational Linguis...

    doi:10.18653/v1/2020.acl-main.421 2020

  3. [3]

    Towards cross-tokenizer distillation: the universal logit distillation loss for llms.arXiv preprint arXiv:2402.12030 , 2024

    Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. Towards cross-tokenizer distillation: the universal logit distillation loss for llms.arXiv preprint arXiv:2402.12030 , 2024

    arXiv 2024

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018

    arXiv 2018

  6. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168 , 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168 , 2021

    arXiv

  7. [7]

    Word translation without parallel data, 2018

    Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data, 2018. URLhttps://arxiv.org/abs/1710.04087

    arXiv 2018

  8. [8]

    Xnli: Evaluating cross-lingual sentence representations.arXiv preprint arXiv:1809.05053, 2018

    Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations.arXiv preprint arXiv:1809.05053, 2018

    arXiv 2018

Show all 36 references
  1. [9]

    FOCUS: Effective embedding initialization for mono- lingual specialization of multilingual models

    Konstantin Dobler and Gerard de Melo. FOCUS: Effective embedding initialization for mono- lingual specialization of multilingual models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023 doi

  2. [10]

    How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings

    Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language P...

    2019 doi

  3. [11]

    A framework for few-shot language model evaluation, 12 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023

  4. [12]

    Fast vocabulary transfer for language model compression

    Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. Fast vocabulary transfer for language model compression. In Yunyao Li and Angeliki Lazaridou, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pa...

    2022 doi

  5. [13]

    Arcee‘s MergeKit: A toolkit for merg- ing large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee‘s MergeKit: A toolkit for merg- ing large language models. In Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimorina, ...

    2024 doi

  6. [14]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783 , 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  7. [15]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  8. [16]

    Initializing new word embeddings for pretrained language models, 2021

    John Hewitt. Initializing new word embeddings for pretrained language models, 2021. URL https:/nlp.stanford.edu/~johnhew//vocab-expansion.html

    2021

  9. [17]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  10. [18]

    Vocab transplantation tool (github repository), 2025

    jukofyork. Vocab transplantation tool (github repository), 2025. URLhttps://github.com/ jukofyork/transplant-vocab

    2025

  11. [19]

    Language models use trigonometry to do addition

    Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition. arXiv preprint arXiv:2502.00873 , 2025

    2025 arXiv

  12. [20]

    Cross-lingual alignment methods for multilingual BERT: A comparative study

    Saurabh Kulshreshtha, Jose Luis Redondo Garcia, and Ching-Yun Chang. Cross-lingual alignment methods for multilingual BERT: A comparative study. In Trevor Cohn, Yulan He, and Yang Liu, editors,Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 933–94...

    2020

  13. [21]

    Shared global and local geometry of language model embeddings, 2025

    Andrew Lee, Melanie Weber, Fernanda Viégas, and Martin Wattenberg. Shared global and local geometry of language model embeddings, 2025. URLhttps://arxiv.org/abs/2503.21073

    2025 arXiv

  14. [22]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning , pages 19274–19286. PMLR, 2023

    2023

  15. [23]

    Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan ...

    2024 arXiv

  16. [24]

    Remya R. K. Menon, S Gargi, and S Samili. Clustering of words using dictionary-learnt word representations. In 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI) , pages 1539–1545, 2016. doi: 10.1109/ICACCI.2016.7732267

    2016

  17. [25]

    Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781 , 2013

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781 , 2013

    2013 arXiv

  18. [26]

    WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational ...

    2022

  19. [27]

    Zero-shot tokenizer transfer, 2024

    Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. Zero-shot tokenizer transfer, 2024. URL https://arxiv.org/abs/2405.07883

    2024

  20. [28]

    Efficient language model training through cross-lingual and progressive transfer learning, 2023

    Malte Ostendorff and Georg Rehm. Efficient language model training through cross-lingual and progressive transfer learning, 2023

    2023

  21. [29]

    The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031 , 2016

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031 , 2016

    2016 arXiv

  22. [30]

    Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition

    Yagyensh Chandra Pati, Ramin Rezaiifar, and Perinkulam Sambamurthy Krishnaprasad. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar conference on signals, systems and computers , pages 40–4...

    1993

  23. [31]

    How good is your tokenizer? on the monolingual performance of multilingual language models

    Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. pages 3118–3135, 01 2021. doi: 10.18653/v1/2021.acl-long.243

    2021 doi

  24. [32]

    Orthogonal matching pursuit for text classification.arXiv preprint arXiv:1807.04715 , 2018

    Konstantinos Skianis, Nikolaos Tziortziotis, and Michalis Vazirgiannis. Orthogonal matching pursuit for text classification.arXiv preprint arXiv:1807.04715 , 2018

    2018 arXiv

  25. [33]

    Signal recovery from random measurements via orthogonal matching pursuit

    Joel A Tropp and Anna C Gilbert. Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on information theory , 53(12):4655–4666, 2007

    2007

  26. [34]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  27. [35]

    Paws-x: A cross-lingual adversarial dataset for paraphrase identification.arXiv preprint arXiv:1908.11828 , 2019

    Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. Paws-x: A cross-lingual adversarial dataset for paraphrase identification.arXiv preprint arXiv:1908.11828 , 2019. 15 A Appendix A.1 Token Decomposition Examples These examples show how tokens are represented as linear co...

    1908 arXiv

  28. [2022]

    URL https://aclanthology.org/2022

    Association for Computational Linguistics. URL https://aclanthology.org/2022. naacl-main.293

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.