{"id":"5dbf86ac-961b-4711-b0d1-f346aadc3dc0","arxiv_id":"2506.06607","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"OMP sparse coding of donor token embeddings, with coefficients transferred to the base embedding space, preserves LLM performance after tokenizer replacement better than published zero-shot baselines, though simple heuristics tie it when token overlap is high.","lead":"This paper replaces a language model's tokenizer without any training by reconstructing new token embeddings as sparse combinations of shared tokens, using orthogonal matching pursuit. It reports that the method preserves model performance better than published zero-shot baselines on two cross-tokenizer tasks, though simple heuristics are competitive when token overlap is high.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is under-supported in the high-overlap Qwen→Llama experiment: OMP-K64, ZeroEmbed, and MeanEmbed are identical to four decimals on MMLU/ARC/GSM8K, within Table 1's standard errors, contradicting 'far better than simple heuristics.'","rationale":"The paper's strongest evidence is the low-overlap Llama→Mistral NeMo experiment, where OMP clearly beats zero and mean initialization and the listed baselines collapse; that result is real and should be credited. The problem is with the universal framing of the central claim. The abstract says OMP preserves performance 'far better than simple heuristics' and that other zero-shot approaches 'degrade significantly,' but the second headline experiment, Qwen→Llama, shows ZeroEmbed and MeanEmbed matching OMP to four decimals on MMLU, ARC-C, and GSM8K. These are also zero-shot heuristics, and Table 1's own standard errors make the differences statistically indistinguishable. This is an internal inconsistency between the stated claim and the reported measurements, not an external disagreement about consensus. I considered the unproved orthogonal-alignment assumption from Section 3.3 as the load-bearing concern; it is real, but the paper explicitly frames that section as conceptual, and the practical central claim is an empirical comparison. The concrete test I propose would settle whether OMP actually separates from simple heuristics in the high-overlap regime, and would determine whether the abstract needs qualification or the comparison needs to be strengthened. The reader's weakest_assumption points to the geometric premise, but the reader's rationale already flags the high-overlap table; hence partial agreement. The conditional verdict remains appropriate: the low-overlap result justifies interest, but the categorical claim needs either stronger significance evidence or narrowing.","tokens_in":13642,"tokens_out":6250,"duration_ms":70282,"concrete_test":"Re-run the Qwen→Llama high-overlap evaluation for OMP-K64, ZeroEmbed, and MeanEmbed using the LM Evaluation Harness with paired bootstrap (sample the same query indices for all methods) on MMLU, ARC-C, and GSM8K, and report 95% confidence intervals. If OMP's intervals overlap those of ZeroEmbed/MeanEmbed on all three benchmarks, then Table 3 does not support the 'far better than simple heuristics' claim, and the headline conclusion should be scoped to low-overlap settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 1 claim OMP preserves model performance 'far better than simple heuristics (zero, mean) or other zero-shot approaches' and that other zero-shot approaches 'degrade significantly.' The low-overlap Llama→Mistral NeMo experiment (Table 2) supports this: OMP-K64 reaches 62.2% MMLU versus 58.5%/59.4% for zero/mean. But in the high-overlap Qwen→Llama experiment (Table 3), the comparison collapses. On MMLU, ARC-C, and GSM8K, OMP-K64, ZeroEmbed, and MeanEmbed are numerically identical: MMLU 0.3640/0.3640/0.3641; ARC 0.3626/0.3626/0.3626; GSM8K 0.0144/0.0144/0.0144. Table 1 reports bootstrap standard errors of 0.004 (MMLU), 0.014 (ARC), and 0.013 (GSM8K); the differences are within those errors, and the errors are omitted from the main tables. OMP is only slightly better on Belebele and WikiText, not 'far better.' Because this is one of only two headline experiments, the paper's categorical claim is not established. The conclusion needs to be scoped to low-overlap settings, or backed by a significance test showing actual separation from simple heuristics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free tokenizer-transplantation method: for each token in the donor vocabulary absent from the base vocabulary, it computes a k-sparse representation of the donor embedding in terms of shared-token donor embeddings using Orthogonal Matching Pursuit, then writes the same coefficients as a linear combination of the corresponding base embeddings. Shared tokens are copied. Experiments transplant Llama 3 into Mistral NeMo 12B and Qwen 2.5 into Llama 3 1B, plus a matched-numeric-scheme control (Mistral NeMo→Qwen 7B), and compare against zero, mean, WECHSEL, FOCUS, ZETT, and CLPTransfer on MMLU, ARC-C, GSM8K, XNLI, AGIEval, LAMBADA, Belebele, and WikiText perplexity. The paper also reports continued-pretraining results and integrates the method into the open-source mergekit-tokensurgeon tool.","tokens_in":13847,"tokens_out":5421,"duration_ms":52023,"significance":"If the empirical claims held uniformly, the method would be a practical, low-cost tool for cross-tokenizer distillation, speculative decoding, and vocabulary expansion, and the paper's release of an efficient incremental-QR OMP implementation plus reproducible harness evaluations would make it directly usable. The Mistral NeMo→Qwen control is a useful falsifiable test of the numerical-tokenization hypothesis, and the finding that matched digit-tokenization schemes reduce GSM8K degradation to -5.6% is a concrete, non-obvious result. The main weakness is that the headline comparative claim is only clearly supported in the low-overlap Llama→Mistral NeMo experiment; in the high-overlap Qwen→Llama experiment the OMP and trivial baselines are indistinguishable on several primary metrics. This does not void the contribution but requires scoping or additional statistical support before the abstract's stronger wording is justified.","major_comments":[{"comment":"The claim that OMP preserves performance 'far better than simple heuristics (zero, mean)' is contradicted by the Qwen→Llama results. In Table 3, OMP-K64, ZeroEmbed, and MeanEmbed produce identical values to four decimals on MMLU (0.3640/0.3640/0.3641), ARC-C (0.3626/0.3626/0.3626), and GSM8K (0.0144/0.0144/0.0144); Table 1 reports bootstrap standard errors of 0.004, 0.014, and 0.013 for these benchmarks, so the observed differences are within noise. Section 4.2's statement that 'This performance outperforms all simpler heuristics' is not supported by these numbers. Please either restrict the comparative claim to low-overlap settings, add significance tests or confidence intervals that demonstrate separation, or explicitly present the high-overlap case as a case where OMP is competitive but not superior.","section":"Abstract; Section 1; Table 3; Table 1"},{"comment":"The transfer of sparse coefficients from donor to base space is justified by 'Approximate Orthogonal Equivalence' plus sparse-reconstruction stability, and the paper explicitly disclaims proving such an alignment. Because this is the load-bearing geometric premise, the manuscript should provide direct empirical evidence: for example, estimate an orthogonal map U on shared tokens and report the alignment error ‖U e^donor_j − e^base_j‖, or compare the OMP reconstruction residual in donor space with the actual error of the transplanted embedding in base space. Without such a check, the reader cannot distinguish a method that works because of the stated geometry from one that works for other reasons (e.g., the shared tokens dominating benchmark inputs).","section":"Section 3.3"},{"comment":"The sparsity level k=64 is selected after scanning the same benchmarks (MMLU, ARC, GSM8K, etc.) used in the main comparison tables, and only the selected value is used in the headline comparisons (Tables 2–4). This makes the reported OMP numbers post-selection and partly optimistic relative to the fixed baselines. Please report results for all k values in the main comparison or choose k on a validation split, and state whether the conclusions change for k=8 or k=16. The effect is likely modest, but it affects the claim that OMP 'consistently achieves the best overall performance'.","section":"Section 4.3"}],"minor_comments":[{"comment":"The abstract contains 'atraining-free' with a missing space.","section":"Abstract"},{"comment":"The sentence begins 'Inspeculative decoding' with a missing space before 'speculative'.","section":"Section 1"},{"comment":"The CLPTransfer WikiText entry of 6791.6044 bits/byte and the corresponding '+ 1e6%' relative change are awkwardly formatted; use scientific notation and consistent rounding.","section":"Table 3"},{"comment":"The sparse decomposition display is a table but is captioned as 'Figure 1'; renumber or recaption it for consistency.","section":"Appendix A.1"},{"comment":"The FOCUS row reports LAMBADA accuracy 0.0000; if this is literally zero correct predictions, state that explicitly, since such an exact zero is surprising and may indicate a degenerate output.","section":"Table 2"},{"comment":"The sentence 'zero-shot OMP alone already outperforms other zero-shot heuristics' is too broad: in Table 5, OMP-K64 zero-shot XNLI (0.3430) is below ZETT (0.3437) and FOCUS (0.3578). Qualify the sentence to say 'overall' or 'on most benchmarks'.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious empirical contribution with a practical tool release, and the low-overlap experiment plus the matched-numeric-tokenization control are valuable. The main issue is that the abstract and Section 1 overstate the results relative to Table 3, where OMP and trivial baselines are statistically indistinguishable on three of five reported benchmarks. I recommend major revision rather than rejection because the central method is defensible and the overclaim can be fixed by scoping and adding significance tests. No citation or authorship concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is sound and the low-overlap experiment is convincing, but the high-overlap experiment does not support the paper's headline claim. That is the main thing to know.\n\nWhat's new: the two-phase OMP coefficient transfer from donor to base embedding space is a clean trick, and as far as I can tell it hasn't been tried for cross-tokenizer initialization. The paper also identifies numeric-tokenization mismatch as a key failure mode, and the control experiment with matched digit schemes (Mistral NeMo→Qwen) supports that story. The code is released in mergekit-tokensurgeon, which makes the method easy to try.\n\nWhat it does well: For Llama→Mistral NeMo, where overlap is only 54%, OMP-K64 clearly beats zero and mean initialization and the published baselines applied zero-shot. That's a real result. The computational cost is also low: 38–74 seconds on an H100, versus hours for CLPTransfer/FOCUS and big meta-training for ZETT.\n\nSoft spots: the Qwen→Llama experiment (86% overlap) contradicts the abstract. On MMLU, ARC-C, and GSM8K, OMP-K64, ZeroEmbed, and MeanEmbed are identical to four decimals, and the tiny differences are within the bootstrap standard errors the paper itself reports. So the sentence 'This performance outperforms all simpler heuristics' in Section 4.2 is just not true for those metrics. The claim that 'other zero-shot approaches degrade significantly' is fine if it refers only to FOCUS/ZETT/WECHSEL/CLPTransfer, but the paper lumps 'simple heuristics' together with them in the introduction, and that's where the overreach is. Also, k is selected after looking at the same benchmarks, which inflates the apparent advantage. That's a solvable problem: fix k, report error bars, and scope the conclusion to low-overlap settings.\n\nThe geometric justification is explicitly disclaimed, which is honest. It's a plausible heuristic, not a theorem. I don't think that's a fatal flaw; the empirical result in the low-overlap case stands on its own.\n\nBottom line: this is a useful practical method with one solid experiment, one weak comparison, and an overbroad abstract. It deserves peer review, but the authors need to fix the overclaim and add error bars. I'd want to cite the low-overlap result; I'd be cautious about citing the high-overlap one. Bring it to reading group if you want a case study in benchmark evaluation.","headline":"A sound low-overlap result and a genuinely useful OMP trick are oversold by a high-overlap experiment that shows OMP is statistically indistinguishable from zero and mean initialization.","tokens_in":14460,"tokens_out":3418,"would_cite":true,"duration_ms":32938,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free method transplants tokenizers between pretrained LLMs by reconstructing unseen token embeddings with Orthogonal Matching Pursuit.","keywords":["tokenizer transplantation","orthogonal matching pursuit","embedding initialization","zero-shot transfer","large language models","sparse approximation","numerical tokenization","vocabulary alignment"],"falsifier":"Take one of the paper's tokenizer pairs and compute a Procrustes orthogonal map between the shared-token embeddings; if the residual on unseen but semantically known tokens is not substantially smaller than the residual of a random orthogonal map, the alignment premise fails. On a more direct level, transplant a new pair of LLMs with mismatched numeric schemes using OMP-K64 and compare its benchmark scores with mean-initialization; a reversal of the reported ordering would refute the claim that OMP best preserves zero-shot performance.","tokens_in":13334,"feed_emoji":"🧩","tokens_out":6208,"duration_ms":61000,"temperature":0.7,"pith_summary":"The paper claims that a pretrained model's tokenizer can be swapped for another model's tokenizer with no gradient updates, preserving most of the original model's performance, by reconstructing each unseen token's embedding as a sparse linear combination of shared anchor tokens. The reconstruction is done in the donor model's embedding space, and the same coefficients are then placed into the base model's embedding space. On Llama-to-Mistral-NeMo (12B) and Qwen-to-Llama (1B) transplants, the paper reports that this beats zero-initialization, mean-initialization, and published zero-shot initialization methods across MMLU, ARC, XNLI, GSM8K, LAMBADA, and perplexity benchmarks. The paper identifies mismatched numerical tokenization schemes (single-digit versus triplet chunking) as the main cause of large drops in mathematical reasoning, and supports this by showing that a transplant between models with matched numeric schemes loses only about five percent on GSM8K. If the method holds up, tokenizer replacement becomes a cheap post hoc operation, enabling cross-tokenizer distillation, speculative decoding, ensembling, and domain vocabulary expansion.","feed_headline":"Sparse greedy matching transplants LLM vocabularies with no retraining","feed_subtitle":"OMP rebuilds unseen token embeddings from shared anchors, beating zero-init, mean-init, and published methods on two LLM pairs.","key_machinery":"The central object is Orthogonal Matching Pursuit (OMP), a greedy sparse-recovery algorithm that iteratively selects the shared-token embedding most correlated with the current residual, solves a least-squares fit on the chosen anchors, and updates the residual. The paper's trick is to run OMP entirely in the donor embedding space and transplant only the signed coefficients into the base embedding space, so the two spaces' dimensionalities never need to match. The geometric justification is approximate orthogonal equivalence: if donor and base embedding spaces are related by a near-orthogonal transformation on the shared vocabulary, then applying that transformation to a sparse donor-space combination yields approximately the same combination in the base space. An incremental QR decomposition of the selected anchor columns keeps per-iteration cost linear, making the transfer practical for vocabularies of roughly one hundred thousand tokens.","core_discovery":"On the paper's own terms, the discovery is that Orthogonal Matching Pursuit gives a training-free way to construct a new vocabulary embedding matrix that is consistent with the base model's embedding geometry, and that this preservation is consistently better across classification, reasoning, and perplexity benchmarks than zero-initialization, mean-initialization, and the existing zero-shot initialization methods tested. The central operation is to approximate each out-of-vocabulary token as a sparse signed combination of shared anchor embeddings in the donor space, then transplant exactly those coefficients into the base space. The paper further claims that most remaining degradation in mathematical reasoning comes from structural mismatches in numerical tokenization rather than from the transplantation mechanism itself, supported by a control experiment with matched single-digit numeric schemes in which GSM8K drops by about 5.6 percent instead of more than seventy percent.","pith_inferences":["A natural extension is to apply the same sparse-coefficient transfer to attention output or feed-forward projection spaces, though those spaces are less isotropic and may not enjoy the same approximate orthogonal alignment.","The numerical-tokenization finding predicts a testable rule: OMP transplants between tokenizers that both use single-digit tokens or both use triplet chunking should show much smaller math-reasoning drops than mixed-scheme pairs; a broad survey across tokenizer pairs would establish how general the rule is.","The use of signed coefficients lets OMP explore a larger linear subspace than convex-hull methods, which may explain its advantage, but it also means anchors with large opposing coefficients deserve scrutiny as potential sources of instability.","A hybrid strategy that applies OMP for general vocabulary while using dedicated initialization for numeric tokens is an explicit, plausible route to recovering most lost mathematical ability, and the paper's control experiment suggests the expected gain is large."],"forward_implications":["Language models can be retokenized in minutes on a single GPU, enabling the direct reuse of pretrained weights under a new tokenizer without continued pretraining.","Teacher and student models with different vocabularies become directly comparable for logit-level distillation, and arbitrary model pairs become usable in speculative decoding without prior vocabulary alignment.","OMP can initialize embeddings for new domain-specific tokens by reconstructing them from existing anchors, adding minimal vocabulary overhead while preserving existing model performance.","When numeric tokenization schemes are matched, mathematical reasoning can survive transplantation nearly intact, as the paper's GSM8K control experiment shows, so tokenizer-pair selection is itself a practical design lever."],"supporting_citations":[{"why":"supplies the Orthogonal Matching Pursuit algorithm used to select a sparse anchor set for each unseen token.","marker":"[30]"},{"why":"supplies the approximate orthogonal equivalence of independently trained embedding spaces used as the geometric justification.","marker":"[7]"},{"why":"provides sparse-recovery guarantees for OMP that motivate reliable reconstruction of unseen token embeddings.","marker":"[33]"},{"why":"supplies evidence of linear and analogical structure in word embeddings, cited as a reason OMP captures semantic directions.","marker":"[25]"},{"why":"supplies the overlapping-token embedding initialization baseline against which OMP is compared zero-shot.","marker":"[26]"},{"why":"supplies a meta-trained zero-shot tokenizer-transfer baseline against which OMP is compared.","marker":"[27]"},{"why":"supplies an overlapping-token weighting initialization baseline against which OMP is compared.","marker":"[9]"},{"why":"supplies a cross-lingual progressive-transfer baseline against which OMP is compared.","marker":"[28]"}],"fun_headline_variants":["OMP rebuilds missing token embeddings without any training","Sparse coding transplants LLM vocabularies, preserving performance","No training needed: OMP matches LLM tokenizers with sparse anchors","OMP sparse reconstruction transplants tokenizers with zero training","Best zero-shot tokenizer swap: OMP sparse anchors beat published methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the donor and base embedding spaces are aligned by an approximately orthogonal transformation on the shared vocabulary; if that alignment does not hold, the transplanted coefficients are not guaranteed to represent the same meaning in the base space.","fun_headline_variants_meta":{"raw":{"variants":["OMP rebuilds missing token embeddings without any training","Sparse coding transplants LLM vocabularies, preserving performance","No training needed: OMP matches LLM tokenizers with sparse anchors","OMP sparse reconstruction transplants tokenizers with zero training","Best zero-shot tokenizer swap: OMP sparse anchors beat published methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2798,"prompt_tokens":975,"completion_tokens":1823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1737}},"tokens_in":591,"tokens_out":1823,"duration_ms":11309,"temperature":1.0,"reasoning_tokens":1737,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:53:06.521481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the paper's tokenizer pairs and compute a Procrustes orthogonal map between the shared-token embeddings; if the residual on unseen but semantically known tokens is not substantially smaller than the residual of a random orthogonal map, the alignment premise fails. On a more direct level, transplant a new pair of LLMs with mismatched numeric schemes using OMP-K64 and compare its benchmark scores with mean-initialization; a reversal of the reported ordering would refute the claim that OMP best preserves zero-shot performance.","supporting_citations":[{"cited_title":"Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition","cited_arxiv_id":null,"evidence_quote":"supplies the Orthogonal Matching Pursuit algorithm used to select a sparse anchor set for each unseen token."},{"cited_title":"Signal recovery from random measurements via orthogonal matching pursuit","cited_arxiv_id":null,"evidence_quote":"provides sparse-recovery guarantees for OMP that motivate reliable reconstruction of unseen token embeddings."},{"cited_title":"WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models","cited_arxiv_id":null,"evidence_quote":"supplies the overlapping-token embedding initialization baseline against which OMP is compared zero-shot."},{"cited_title":"Efficient language model training through cross-lingual and progressive transfer learning, 2023","cited_arxiv_id":null,"evidence_quote":"supplies a cross-lingual progressive-transfer baseline against which OMP is compared."}],"review_version":1}