{"id":"09d1e392-038b-474e-bcef-d8c84be85f88","arxiv_id":"2504.17025","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SAVA learns a linear map from Minerva's Italian embedding space to reinitialize Llama and Mistral vocabularies, cutting token fertility by up to 25% and Llama's parameters by 1B while nearly preserving Italian task performance after 12B training tokens.","lead":"The authors replace the tokenizers of English-trained LLMs with an Italian tokenizer and re-initialize the embeddings using a learned linear map, a method they call SAVA. This cuts token counts by up to 25% and shrinks Llama-3.1-8B by about 1 billion parameters, while Italian task performance is nearly recovered after continued training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SAVA's core step extrapolates a linear map from 16k-20k shared tokens to all Italian-only subwords; if that extrapolation fails, the method's claimed semantic-alignment advantage is unsupported, though 12B-token recovery is still empirically shown.","rationale":"Read as an empirical methods paper, the reported experiments do support the headline that replacing the tokenizer and training for about 12B tokens yields competitive Italian performance, with SAVA at least comparable to FVT and LAPT on the shown tables. The release of code and models makes the proposed test feasible, which is a point in the paper's favor. The real soft spot is the mechanism: SAVA learns a linear map on shared tokens and applies it to all Italian-only subwords without any direct evaluation of that extrapolation. This is not an internal inconsistency, but it is load-bearing for the claim that SAVA's semantic alignment, rather than generic continual training, drives the result. The paper's own evidence partially mitigates the concern: SAVA shows lower initial loss than CLP and Random in Figures 6-7, and Table 7 indicates closer aggregate similarity to Minerva-3B. However, these measurements do not isolate Vt\\Vs, and the shared-token subset is not representative of the full target vocabulary. The conditional verdict remains appropriate: the empirical recovery claim stands, but the distinct contribution of SAVA should be verified by a held-out or token-type-specific test, or the paper should be reframed to avoid overstating the mechanism.","tokens_in":17475,"tokens_out":9366,"duration_ms":100689,"concrete_test":"Using the released code and models, train phi on a random 50% of Vs∩Vt and evaluate cosine similarity and MSE on the held-out half; then apply the same diagnostics to tokens in Vt\\Vs, e.g. distance from phi(Eh(t)) to the nearest source embedding among shared tokens. If held-out shared-token error is much larger than training error, or if out-of-vocabulary mapped embeddings fall outside the shared-token distance distribution, the extrapolation in Eq. (1) is unsupported. A complementary behavioral check: at step 0, compute Italian cross-entropy separately for tokens in Vs∩Vt and Vt\\Vs; if SAVA's advantage over FVT and Random is confined to shared tokens, its claimed benefit for new Italian subwords is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 3.1, Eq. (1): phi is fit on tokens in Vs∩Vt (16,438 for Mistral, 20,358 for Llama) and then applied to every token in Vt\\Vs. The central claim that SAVA-initialized models recover quickly depends on those unseen Italian subwords receiving sensible source-space embeddings. The paper provides no direct check of this extrapolation: Appendix B only varies the number of training tokens and the helper size, and Section 6 measures aggregate embedding similarity over 256 anchor tokens, not per-token alignment for Vt\\Vs. Shared tokens are not a random sample of the target vocabulary; they are exactly the subwords two tokenizers happen to agree on, often punctuation, ASCII fragments, and English or common subwords, while Italian-only tokens are typically full words and morphological forms. If the linear relation estimated on shared tokens does not hold for these out-of-vocabulary forms, the initial SAVA embeddings for the Italian-specific part of the vocabulary may be poorly conditioned, and the observed fast convergence and final scores would be carried by the subsequent 12B-token continual training rather than by SAVA's semantic alignment. Table 7's higher aggregate similarity to Minerva-3B does not resolve this, since it could reflect over-attachment to the helper space rather than correct placement in the source space. This does not falsify the empirical recovery results, but it removes the mechanism that justifies SAVA as a contribution over simpler baselines such as FVT.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the inefficiency of English-centric LLM tokenizers on Italian by comparing vocabulary adaptation methods and introducing SAVA, which replaces the source tokenizer with Minerva's Italian tokenizer and initializes target embeddings by learning a linear map from Minerva-3B's embedding space to the source model's embedding space. The authors apply the methods to Mistral-7B-v0.1 and Llama-3.1-8B, report fertility reductions of 25% and 16%, and a 1B-parameter reduction for Llama, then continue training on 12B tokens of mixed Italian/English data. They evaluate on translated multiple-choice benchmarks (ITA-Bench), FLoRes, and SQuAD-it, and analyze the resulting embedding structure. The central claim is that SAVA-initialized models recover base-model performance after about 12B tokens and converge faster than other initialization heuristics.","tokens_in":17745,"tokens_out":7507,"duration_ms":65177,"significance":"The paper's empirical scope is a strength: two 7-8B models, a fixed 12B-token budget, a consistent evaluation harness, publicly released code and models, and an environmental-impact statement. If the results hold, vocabulary substitution plus continual training is a practical alternative to full Italian continual pretraining. The paper also provides a useful comparison of FVT, CLP, Random, and LAPT. However, the incremental advantage of SAVA over the much simpler FVT is small and not statistically established, and the extrapolation at the core of SAVA is untested. The empirical recovery claim is well supported; the novel-method claim needs strengthening.","major_comments":[{"comment":"The load-bearing assumption of SAVA is that the linear map phi learned on the shared vocabulary Vs intersection Vt transfers to all tokens in Vt without Vs, since those tokens are initialized solely through phi. The paper does not test this extrapolation directly. Appendix B varies the amount of training data and the helper model size but only measures the continual-training loss, and Section 6 uses 256 anchor tokens and reports an aggregate relative-representation similarity, which does not isolate the quality of the out-of-vocabulary token embeddings. Because the shared tokens may be systematically different from Italian-only subwords (e.g., punctuation, ASCII fragments, and frequent English tokens), the claim that SAVA provides semantically aligned initializations for Italian-specific tokens remains unverified. I recommend adding a diagnostic: measure the mapping error on a held-out split of shared tokens, or compare nearest neighbors or zero-shot perplexity restricted to the Vt without Vs tokens against an independent bilingual lexicon.","section":"Section 3.1, Eq. (1)"},{"comment":"The paper's central methodological contribution, SAVA, is not statistically distinguishable from FVT in the reported results. In Table 3 at 2000 steps, SAVA averages 66.99 versus 66.94 for FVT, and in Table 4, SAVA averages 66.29 versus 65.79 for FVT; per-task standard errors are roughly 0.4 to 1.2 points, and no multiple seeds or significance tests are reported. The text nonetheless claims that SAVA consistently achieves higher overall scores and faster convergence (Figures 2 and 3). Without a significance test, multiple independent runs, or a larger effect, the comparison only supports the weaker claim that SAVA is competitive with FVT. The conclusions and abstract should be reworded accordingly, or the missing evidence should be added.","section":"Tables 3 and 4; Section 5.1.1"},{"comment":"The statement that for Mistral-7B-v0.1 'we do not increase its vocabulary size or model parameters' is contradicted by Table 1, which reports 32,768 tokens versus 32,000 and 7.25B parameters versus 7.24B. This is a stated contribution and should be corrected, or the wording should be changed to describe a negligible increase rather than no increase.","section":"Abstract and Introduction; Table 1"},{"comment":"The claim that 'for both models the SAVA approach leads the model to achieve slightly higher performance in the source language' is not supported for Llama-3.1-8B: Table 11 shows SAVA at 75.33 average versus LAPT at 77.29 at 3000 steps, so LAPT is substantially higher. The sentence should be qualified to refer only to the vocabulary-adapted methods, or removed.","section":"Section 5.1.2; Table 11"}],"minor_comments":[{"comment":"The conclusion that 'Mistral-7B-v0.1 reached base model performance after processing 2 billion tokens' is not tied to a specific table; the 400-batch checkpoint (approximately 2.5B tokens) is not reported in the tables, so please add the corresponding values or cite the relevant figure more precisely.","section":"Section 5.1.1 and Section 7"},{"comment":"The text says the Minerva tokenizer has 'on average 25% of fertility gain' compared to Mistral on Italian text, but the two reductions in Table 2 are about 26% on CulturaX and 19% on Wikipedia; please state the exact computation or report the numbers as 'up to 26%'.","section":"Table 2 and Section 4.1"},{"comment":"The model name is written inconsistently as 'LLaMa-3-8B' in Table 2 and 'Llama-3.1-8B' elsewhere; please use one consistent name throughout.","section":"Tables 2 and 4"},{"comment":"The sentence 'SAVA and FVT achieve results at 400 batches that are comparable to those of the Random approach at the end of training' refers to a checkpoint that does not appear in the tables; please include the 400-batch numbers or state the corresponding token count explicitly.","section":"Section 5.1.1"},{"comment":"The description of the anchor selection as '128 non-prefix tokens and 128 prefix tokens' is ambiguous, since 'prefix' could mean tokens that begin a word rather than model prefix tokens; please clarify the terminology.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid, honest empirical comparison of vocabulary adaptation methods for Italian, with a new method (SAVA) that learns a linear map from a helper model's embedding space to the source model's. The headline result—swap the tokenizer, initialize embeddings, continually train on ~12B tokens, and you recover base-model performance with 25% lower fertility for Mistral-7B and 16% for Llama-3.1-8B, plus a 10% parameter cut for Llama—is supported by the tables. That is a genuinely useful recipe for making English-centric LLMs cheaper and faster for Italian.\n\nWhat's actually new: applying latent-space linear mapping to vocabulary substitution for decoder-only LLMs, and the systematic comparison with FVT, CLP, Random, and LAPT on two models. The experimental design is careful: fixed compute budget, same data mix, loss trajectories, embedding-similarity analysis, and they release code and models. The loss curves show SAVA helps early convergence, and the embedding analysis shows SAVA aligns more closely with the helper space.\n\nThe soft spots are real but not fatal. SAVA's final advantage over FVT is small, often within reported standard errors. The mechanism depends on extrapolating the linear map from 16k–20k shared tokens to all Italian-only subwords, and the paper doesn't directly test that extrapolation. Shared tokens are likely unrepresentative—ASCII fragments, punctuation, common words—so the stress-test concern is legitimate. However, the empirical recovery claim doesn't rest on SAVA's mechanism being right; FVT recovers too, and SAVA's benefit shows up early in training. Missing strong baselines (WECHSEL, OFA, FOCUS) is a gap, and the reliance on automatically translated ITA-Bench plus two generative tasks limits precision. The abstract also over-attributes the fertility/efficiency gains to SAVA when they come from the tokenizer swap.\n\nWho is this for: anyone working on multilingual adaptation or efficient LLM serving. It deserves a serious referee. I'd recommend engaging with it, with the expectation that reviewers ask for stronger baselines and a direct check of the extrapolation step—e.g., per-token alignment quality on held-out Italian subwords.","headline":"Solid empirical recipe for Italian vocabulary adaptation; SAVA is a reasonable new heuristic, but its advantage over simpler FVT is thin and the extrapolation step is under-validated.","tokens_in":18387,"tokens_out":2536,"would_cite":true,"duration_ms":21281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new method, SAVA, transplants an Italian tokenizer into English LLMs, cutting Italian token use by 25% and restoring performance after 12B training tokens.","keywords":["SAVA","Semantic Alignment Vocabulary Adaptation","vocabulary adaptation","token fertility","Italian language models","continual training","embedding initialization","tokenizer substitution"],"falsifier":"Apply SAVA to a target language whose subword vocabulary barely overlaps with the source model's (e.g., Finnish), using the same 12B-token continual-training budget. If the out-of-vocabulary embeddings produced by the shared-token map do not lower the initial training loss or final downstream scores relative to FVT and random initialization, the assumption that the linear relation extrapolates to unseen subwords is wrong.","tokens_in":17223,"feed_emoji":"🇮🇹","tokens_out":16009,"duration_ms":127429,"temperature":0.7,"pith_summary":"English-heavy LLMs encode Italian inefficiently: they split words into more tokens than an Italian-native tokenizer would, and pay for it in slower inference, larger memory, and a bigger vocabulary than the language needs. This paper proposes Semantic Alignment Vocabulary Adaptation (SAVA), which replaces the English tokenizer and vocabulary of Mistral-7B-v0.1 and Llama-3.1-8B with Minerva-LLMs' Italian tokenizer, and initializes the new embeddings with an affine map $\\phi(x)=Wx+b$ learned from the tokens the two vocabularies share. The reported result is that token fertility on Italian text falls by 25% for Mistral and 16% for Llama, Llama loses about one billion parameters, and after about 12 billion tokens of continual training on mixed Italian-English data the adapted models match or approach the base models and a language-adaptive pretraining baseline on Italian benchmarks. If the paper is right, SAVA is a cheaper route to Italian-capable LLMs than continued pretraining with the original English tokenizer, with faster convergence and lower inference cost.","feed_headline":"Italian tokenizer swap cuts English LLM token use by 25%","feed_subtitle":"SAVA transplants the Italian vocabulary, shrinks Llama by a billion parameters, and recovers performance in 12B tokens.","key_machinery":"The central mechanism is Semantic Alignment Vocabulary Adaptation (SAVA), an embedding-initialization procedure built on the premise that two LLM embedding spaces are related by an affine map. For every token $t_i$ in the shared vocabulary $V_s \\cap V_t$, SAVA has two representations: $E^{t_i}_s$ in the English model and $E^{t_i}_h$ in the helper model Minerva-3B. It fits a single-layer feed-forward network $\\phi(x)=Wx+b$ by minimizing the squared error between $W E^{t_i}_h + b$ and $E^{t_i}_s$ over the shared tokens, then initializes each Italian-only token $t_i \\in V_t \\setminus V_s$ as $\\phi(E^{t_i}_h)$. This replaces bilingual-dictionary or averaged-sub-token heuristics with one learned projection, and the paper's loss curves and downstream scores attribute SAVA's faster convergence and closer structural alignment to the helper model to this initialization. Fertility, the average number of tokens into which a word is split, is the efficiency quantity that motivates the whole substitution.","core_discovery":"On its own terms, the paper establishes that vocabulary substitution for decoder-only LLMs can be made data-efficient by aligning embedding spaces before training. SAVA keeps the source model's weights, swaps the tokenizer and vocabulary for Minerva-LLMs' Italian tokenizer, and initializes every new Italian token's embedding by applying the learned map $\\phi$ to that token's representation in Minerva-3B. The affine map is trained with mean squared error on the 16,438 tokens that Mistral and Minerva share (20,358 for Llama), and the same map is applied to the Italian-only tokens the source model never saw. After about 12 billion tokens of continual training on 75% Italian and 25% English text, the adapted models match or approach the scores of LAPT and, on several tasks, the unmodified base models, while SAVA's training loss is lower from the outset and its embedding structure is closer to Minerva-3B than any other heuristic tested.","pith_inferences":["The paper demonstrates SAVA only for Italian, but the mechanism is language-agnostic; the same linear-map recipe should work for any target language that has a native tokenizer and a smaller helper model.","The ablations show that helper-model size (350M–3B) changes the loss trajectory less than the number of shared tokens, suggesting that even a small, well-matched helper tokenizer could suffice for low-resource languages.","Because the vocabulary replacement alone removes a large share of Llama's embedding parameters, the technique could be combined with quantization or layer pruning to obtain memory savings well beyond the reported 10%.","A stronger helper model than Minerva-3B could improve SAVA further; the paper notes Minerva-7B was not available at the time of writing."],"forward_implications":["Italian text is encoded with 25% fewer tokens for Mistral-7B and 16% fewer for Llama-3.1-8B, which translates directly into shorter inference time and lower serving cost.","The Llama-3.1-8B adaptation removes roughly 75% of the original vocabulary and about one billion parameters, so the resulting model is 10% smaller and faster to load.","SAVA-adapted models converge faster during continual training, matching at 400 batches what random initialization reaches only at the end of the 12B-token budget.","After continual training, vocabulary-adapted models reach scores comparable to LAPT on the Italian multiple-choice benchmarks and generative tasks, while retaining slightly better English performance among the adapted variants.","Mistral's SAVA adaptation reaches base-model performance after only about 2 billion tokens, indicating the 12B-token budget is not a hard requirement."],"supporting_citations":[{"why":"It is the source model whose tokenizer and embeddings are replaced; its Italian fertility is the baseline reduced by 25%.","marker":"Jiang et al., 2023"},{"why":"It is the source model whose 128k-token vocabulary is replaced by Minerva's, removing about one billion parameters.","marker":"Dubey et al., 2024"},{"why":"It supplies the Minerva-LLMs family, the Italian tokenizer substituted into both source models, and the Minerva-3B helper embeddings used by SAVA and CLP.","marker":"Orlando et al., 2024"},{"why":"It provides the theoretical grounding that embedding spaces of different models are related by a linear or conformal map, motivating SAVA's affine mapping.","marker":"Maiorca et al., 2024"},{"why":"It is earlier vocabulary-adaptation work that learns a linear mapping between embedding spaces, and SAVA extends this idea from BERT-style encoders to decoder-only LLMs.","marker":"Koto et al., 2021"},{"why":"It defines the FVT baseline that averages source sub-token embeddings, the main heuristic SAVA is compared against.","marker":"Gee et al., 2022"},{"why":"It defines the CLP baseline using cosine-similarity weighting in the helper embedding space, another principal comparator.","marker":"Ostendorff and Rehm, 2023"},{"why":"It provides the 75% Italian / 25% English data-mixing recipe used in continual training.","marker":"Csaki et al., 2024"},{"why":"It provides CulturaX, the cleaned multilingual corpus used for the continual-training stage.","marker":"Nguyen et al., 2024"}],"fun_headline_variants":["SAVA swaps tokenizer, cuts Italian LLM token use 25%","SAVA reduces Mistral token fertility 25%, Llama params by 1B","SAVA: 25% fewer Mistral tokens, 1B fewer Llama parameters","SAVA aligns embeddings to swap vocab for Italian LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"SAVA's success rests on the assumption that the affine relationship learned from the tokens the English and Italian models share also holds for Italian subwords that exist only in the new vocabulary.","fun_headline_variants_meta":{"raw":{"variants":["SAVA swaps tokenizer, cuts Italian LLM token use 25%","SAVA reduces Mistral token fertility 25%, Llama params by 1B","SAVA: 25% fewer Mistral tokens, 1B fewer Llama parameters","SAVA aligns embeddings to swap vocab for Italian LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001973,"raw_usage":{"total_tokens":7713,"prompt_tokens":958,"completion_tokens":6755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":6671}},"tokens_in":574,"tokens_out":6755,"duration_ms":47306,"temperature":1.0,"reasoning_tokens":6671,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:51:20.691506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply SAVA to a target language whose subword vocabulary barely overlaps with the source model's (e.g., Finnish), using the same 12B-token continual-training budget. If the out-of-vocabulary embeddings produced by the shared-token map do not lower the initial training loss or final downstream scores relative to FVT and random initialization, the assumption that the linear relation extrapolates to unseen subwords is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the Minerva-LLMs family, the Italian tokenizer substituted into both source models, and the Minerva-3B helper embeddings used by SAVA and CLP."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the theoretical grounding that embedding spaces of different models are related by a linear or conformal map, motivating SAVA's affine mapping."}],"review_version":1}