{"id":"5b7cd38d-2f0b-4d69-b8a7-957601492e8d","arxiv_id":"2505.10945","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SALT initializes target-language embeddings in an LLM by linearly mapping embeddings from a target-language PLM into the LLM's space using semantically similar shared tokens.","lead":"This paper presents SALT, a method that borrows word embeddings from language-specific pretrained models and projects them into a large language model's embedding space using per-token linear regressions. In experiments on German, Arabic, and Vietnamese, SALT outperforms existing embedding-initialization baselines on several benchmarks after language-adaptive pre-training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-token least-squares map (Eq. 3) is rank-deficient when Sparsemax selects fewer anchors than the embedding dimension, projecting non-shared PLM embeddings onto a low-dimensional anchor span; the paper reports neither k nor projection residuals.","rationale":"I agree with the reader's conditional verdict: the empirical claims are not fully supported without error bars, a direct PLM-reuse baseline, and a check of the regression's assumptions. My concern sharpens the reader's weakest assumption rather than replacing it. The reader flagged that per-token least-squares mapping fitted on shared vocabulary must generalize to non-shared tokens; the specific failure mode I identify is rank deficiency: when Sparsemax selects fewer anchors than the embedding dimension, the pseudo-inverse solution is underdetermined and acts as a projection onto the anchor span. This is a concrete, internally checkable property of Eq. 3, not a matter of external consensus. If the projection residual is small, the concern is resolved and the method's mechanism is plausible; if it is large, SALT is not actually transferring the PLM embeddings but only a low-rank slice of them, which would undercut the central claim. I do not see this as requiring a verdict change: the paper should remain conditional pending this diagnostic. The limitations section's acknowledgment of low coverage in Arabic makes the rank check especially relevant, and the absence of reported k values is a gap the authors can close without new experiments beyond a code-level diagnostic.","tokens_in":16515,"tokens_out":6671,"duration_ms":74181,"concrete_test":"Using the authors' code and hyperparameters, instrument Step 4 to record, for every non-shared token in each language/model, the Sparsemax-selected k and the relative projection residual ||e_ti - e_ti P_ti||/||e_ti|| where P_ti = E'_ti^+ E'_ti. If the median k is below ht or the median residual is large (e.g., >0.3), the transfer mechanism is rank-deficient. As a control, fit one global least-squares map on all shared tokens, apply it to all non-shared tokens, and re-run Table 1; if global-map performance is within noise of per-token SALT, the per-token regression is not the active ingredient and the method reduces to a global linear assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing risk is in Step 4 (Eq. 3). For each non-shared token v_ti, a matrix X_ti in R^{ht×hs} is fitted to map the k selected shared-token PLM embeddings E'_ti to the corresponding source-LLM embeddings E'_si. When k < ht, the pseudo-inverse solution X_ti = E'_ti^+ E'_si has rank at most k, so applying it to any PLM embedding first projects that embedding onto the row space of E'_ti (dimension ≤ k) and discards the orthogonal complement. The paper never reports k after Sparsemax or checks the size of the discarded component; with low coverage (Appendix A Table 6, e.g., Arabic Gemma 15.6%) and the sparsity of Sparsemax, k < ht is plausible for many tokens. If so, the method is not 'recycling PLM embeddings' but transferring only their projection onto a small anchor subspace, and the claimed semantic richness of the target PLM is partially lost before adaptation. This directly threatens the mechanism behind Tables 1–3.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SALT, a cross-lingual transfer method that replaces an English-centric LLM's embedding matrix with a target-language PLM's vocabulary, initializing each non-shared token embedding via a per-token linear least-squares map fitted on Sparsemax-selected shared tokens using fastText similarities. The authors evaluate SALT for German, Arabic, and Vietnamese on knowledge-based benchmarks (ARC, HellaSwag, TruthfulQA), machine reading comprehension (MLQA, XQuAD, Belebele), cross-lingual QA, and language-modeling loss during language-adaptive continual pre-training, using Gemma-2b and XGLM-1.7b as source models and BERT/GPT/T5-based PLMs as target models. They report that SALT outperforms FOCUS, OFA, and multivariate initialization on most averaged benchmarks, converges faster, and better preserves English capability.","tokens_in":16737,"tokens_out":7676,"duration_ms":77466,"significance":"If the empirical claims hold, SALT is a useful and lightweight contribution to cross-lingual LLM transfer: it recycles widely available PLMs, requires no architectural changes to the source LLM beyond embedding and head replacement, and yields both faster convergence and parameter savings. The method is clearly specified, the baselines are appropriate, and the experimental design includes multiple languages, source models, and target architectures, with care taken to use the same external embeddings and tokenizers across methods. The paper also provides honest limitations, including the dependency on available target-language PLMs and the lack of instruction-tuning evaluation. However, the central claim of significant outperformance is not yet backed by statistical evidence, and the key linear-projection step has an unexamined rank-deficiency risk that bears directly on whether the method actually transmits the full representational content of the PLM embeddings.","major_comments":[{"comment":"The least-squares solution X_ti = pinv(E'_ti) E'_si defines a linear map whose rank is at most k, the number of Sparsemax-selected shared anchors. When k < h_t (the target PLM hidden size), applying X_ti to a non-shared PLM embedding first projects that embedding onto the row space of E'_ti and discards the orthogonal complement; the transferred embedding then cannot carry the full representational content of the PLM. The manuscript reports neither the distribution of k after Sparsemax, nor the numerical rank of E'_ti, nor the residuals of the fitted regressions. This is especially pertinent for Arabic with Gemma, where Table 6 shows only 15.6% shared-vocabulary coverage, so a large fraction of non-shared tokens are transferred under conditions where k may be small. Please report these diagnostics and, if rank deficiency is prevalent, discuss or implement a regularized estimator (e.g., ridge) or a lower-rank constrained projection.","section":"Section 3, Step 4 (Eq. 3)"},{"comment":"The abstract and Section 5 state that SALT 'significantly outperforms' baselines, but the paper provides no error bars, no multiple-seed results, and no significance tests. The reported averages favor SALT on most language/task combinations, yet individual cells are at parity or below baselines; for example, Table 1 shows XGLM Arabic TruthfulQA with SALT at 28.07 versus FOCUS at 28.72, and Gemma Arabic TruthfulQA with SALT tying Multivariate and FOCUS at 28.98. Without measures of variance, the robustness of the claimed gains and the ranking of methods remain unclear. Please add standard deviations over at least three random seeds for the main tables, and a paired test (e.g., bootstrap over language/task cells) for the aggregate comparisons.","section":"Section 5 (Tables 1-3)"},{"comment":"The MRC and cross-lingual results are reported only for Gemma as the source model. The paper's broader conclusions that SALT 'preserves English capability' and improves cross-lingual understanding are therefore not verified for XGLM, even though Table 1 suggests the benefits may be smaller or more variable there. Please either add corresponding XGLM results for Tables 2 and 3 or explicitly restrict the cross-lingual and MRC claims to the Gemma setting.","section":"Section 5.3 (Tables 2 and 3)"}],"minor_comments":[{"comment":"The handling of target tokens absent from fastText is underspecified: it is unclear whether such tokens receive the random initialization and are then excluded from the regression, or whether they are mapped through a fallback similarity. This matters because the coverage fractions in Table 6 implicitly depend on how these tokens are treated.","section":"Section 3, Step 1"},{"comment":"Table 5 reports evaluation loss at various percentages of total training steps, but the evaluation set (whether the validation split from Appendix C or another corpus) is not specified; please clarify.","section":"Section 4.4 / Table 5"},{"comment":"The loss curves are reported without any smoothing or reproducibility details (e.g., which random seed, whether curves are averaged), and Appendix D reports only the final training loss; providing per-step numerical values or a reproducible plotting script would strengthen the convergence claim.","section":"Figure 2"},{"comment":"The statement that 'In principle, SALT can also be applied to larger models (>7b)' is speculative and unsupported by experiments; consider softening it or citing evidence from related vocabulary-transfer work.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper falls within the journal's scope and the core idea is worth pursuing. The main technical risk is the unexamined rank-deficiency of the per-token least-squares projection, which the authors must address empirically before the method's mechanism can be accepted. I also suggest encouraging the authors to release code and per-seed results, given that the paper's headline claim is an empirical superiority that currently lacks statistical support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know about this paper: SALT is a clean and genuinely new initialization trick for cross-lingual transfer. Instead of recombining source embeddings like FOCUS or OFA, it fits a per-token least-squares map from a target-language PLM's embedding space into the source LLM's space, using sparsemax-selected shared tokens as anchors. That is a real departure, and the empirical picture is mostly favorable: on knowledge benchmarks and MRC, SALT beats the three baselines on most averages, converges to lower CLM loss, and holds up better on English evaluations after target-language training. The architecture-extension results (BERT, T5 as target PLMs) are a useful bonus. If you do language adaptation in practice, this is a plausible compute-saving initialization.\n\nNow the soft spots. The load-bearing one is the regression in Eq. 3. For each non-shared token, the map is fitted on k shared-token pairs. When k is smaller than the PLM hidden dimension—which is almost certainly the case, given sparsemax and the low coverage numbers in Table 6—the least-squares solution has rank at most k. Applying it projects the PLM embedding onto the span of the few selected anchors and discards the orthogonal complement. The paper never reports k, residuals, or any check of how much of the PLM embedding survives. That matters because the entire motivation is 'recycling the rich PLM representations'; as written, the method might be transferring only a low-dimensional slice. This is fixable—report k distributions, use a regularized/ridge fit, or compare against a single global linear map—but it needs to be addressed before the mechanism claim holds.\n\nThe other issues are the usual suspects. No error bars, no multiple seeds, and several individual cells in Tables 1 and 2 where SALT is at parity or below a baseline; the abstract's 'significantly outperforms' is not supported by any statistical test. There is also no ablation that isolates the per-token regression: a baseline that just copies the PLM embedding for non-shared tokens (with no projection) would tell you how much the linear map actually adds. The baselines all start from source embeddings, so they don't separate the PLM's contribution from SALT's projection.\n\nNone of this is fatal. The method is well-specified, the results are consistent enough to be interesting, and the missing analyses are straightforward to produce. This paper deserves peer review; I would send it out with the expectation that the authors add the rank analysis, the direct-reuse baseline, and at least one additional seed.","headline":"SALT is a clean, novel initialization trick with mostly positive results, but the per-token regression is likely rank-deficient and the 'significantly outperforms' claim needs more statistical and ablative support.","tokens_in":17256,"tokens_out":4476,"would_cite":true,"duration_ms":43173,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SALT recycles target-language PLM embeddings to outperform existing cross-lingual transfer methods on German, Arabic, and Vietnamese.","keywords":["SALT","cross-lingual transfer","embedding initialization","pre-trained language models","linear least squares transfer","vocabulary replacement","language adaptation","multilingual LLMs"],"falsifier":"Take a language pair where the source and target vocabularies share very few tokens, fit SALT on those anchors, and compare the resulting model's downstream accuracy and final training loss to multivariate initialization under identical training; if SALT does not improve, the per-token regression did not generalize beyond its anchors.","tokens_in":1766,"feed_emoji":"🌐","tokens_out":2839,"duration_ms":97460,"temperature":0.7,"pith_summary":"This paper proposes SALT, a way to move an English-centric large language model (LLM) into a target language by recycling the embeddings of smaller, target-language pretrained models such as BERT, GPT, or T5. For every token the two vocabularies do not share, SALT finds the target token's nearest neighbors among shared tokens using fastText similarities, then fits a per-token least-squares linear map from the pretrained language model's embedding space into the LLM's embedding space. The paper argues that this preserves the target language's semantic richness while placing new embeddings where the LLM expects them, so the transferred model adapts faster, reaches lower language-modeling loss, and scores higher on downstream tasks than the FOCUS, OFA, and multivariate baselines. The claim matters because it offers a parameter-light, initialization-only route to convert large English-centric models into language-specific ones without training embeddings from scratch.","feed_headline":"Recycled PLM embeddings beat cross-lingual transfer baselines","feed_subtitle":"Per-token linear maps carry small PLM embeddings into English-centric LLMs, improving German, Arabic, and Vietnamese tasks.","key_machinery":"The central object is a per-token linear least-squares transform. For each non-shared target token $v_{t_i}$, SALT uses fastText cosine similarity to rank shared vocabulary tokens, applies Sparsemax to select the dynamic top-$k$ nearest neighbors, and stacks their PLM embeddings $E'_{t_i}$ and source-LLM embeddings $E'_{s_i}$. It then solves $\\arg\\min_{X\\in\\mathbb{R}^{h_t\\times h_s}}\\|E'_{t_i}X_{t_i}-E'_{s_i}\\|$, whose closed-form solution is $X_{t_i}=E'^{+}_{t_i}E'_{s_i}$ via the pseudo-inverse. This unique map per token projects that token's PLM embedding into the LLM's embedding space, preserving target-language semantic structure while aligning with the source model's learned representation space.","core_discovery":"The paper's central claim is that recycling target-language pretrained-language-model (PLM) embeddings through per-token linear regression gives a better initialization for cross-lingual transfer than reweighting or resampling the source LLM's own embeddings. In the reported experiments, SALT achieves the highest average accuracy on knowledge benchmarks (ARC, HellaSwag, TruthfulQA) and reading-comprehension tasks (MLQA, XQuAD, Belebele) across German, Arabic, and Vietnamese for both Gemma-2b and XGLM-1.7b source models. Models initialized with SALT also converge faster and end with lower causal language modeling loss during language-adaptive continual pretraining, and they retain English question-answering ability better than the baselines. The paper further shows that the PLM supplying the embeddings does not have to be a decoder, since BERT and T5 target models also perform competitively, indicating that older small models remain useful embedding sources for contemporary LLMs.","pith_inferences":["The method's success likely depends on vocabulary coverage between the source and target models; the paper's own Arabic results, with the lowest overlap, suggest a testable threshold below which the per-token regression may fail to generalize.","Because SALT only changes initialization, its benefits could compound with instruction tuning or longer continued pretraining, a direction the paper leaves open.","The same per-token regression logic could be applied to intermediate layers or to contextualized PLM representations, potentially transferring deeper target-language knowledge rather than only static embeddings.","Languages without a dedicated target PLM would be out of scope, so a practical extension would need to synthesize or approximate a target embedding space from multilingual PLMs or from the source LLM itself."],"forward_implications":["SALT-initialized models reach the lowest final training loss and converge faster than FOCUS, OFA, and multivariate initialization in every language tested.","SALT improves target-language performance on knowledge-based benchmarks and machine reading comprehension, with the largest gains on HellaSwag and generation-based MRC tasks.","SALT preserves English ability better than the baselines and improves both English-to-target and target-to-English question answering, indicating better inner alignment between languages.","SALT works with encoder, decoder, and encoder-decoder PLMs as embedding sources, so existing small models can be recycled for language transfer.","Replacing the source vocabulary with a target PLM's vocabulary reduces tokenized sequence lengths and total parameters, yielding computational savings during training and inference."],"supporting_citations":[{"why":"Supplies the FOCUS baseline, a weighted-mean embedding initialization from source embeddings that SALT must outperform.","marker":"(Dobler and De Melo, 2023)"},{"why":"Supplies the OFA baseline, a factorization-based embedding initialization method for unseen subword embeddings.","marker":"(Liu et al., 2024)"},{"why":"Supplies the fastText static embeddings used to estimate semantic similarity between shared and non-shared vocabulary.","marker":"(Bojanowski et al., 2017)"},{"why":"Supplies Sparsemax, which selects the dynamic set of nearest shared tokens for each non-shared token.","marker":"(Martins and Astudillo, 2016)"},{"why":"Supplies the pseudo-inverse solution that solves the per-token least-squares transform.","marker":"(Peters and Wilkinson, 1970)"},{"why":"Supplies the BERT architecture target PLM whose embeddings SALT recycles.","marker":"(Devlin et al., 2019)"},{"why":"Supplies the Gemma-2b source LLM used in the transfer experiments.","marker":"(Team et al., 2024)"},{"why":"Supplies the XGLM-1.7b source LLM used in the transfer experiments.","marker":"(Lin et al., 2022b)"},{"why":"Supplies the multilingual knowledge benchmarks used to evaluate non-English downstream performance.","marker":"Lai et al. (2023b)"}],"fun_headline_variants":["SALT recycles PLM embeddings for faster, stronger cross-lingual transfer","Per-token linear maps beat embedding swaps in cross-lingual LLM transfer","Recycled PLM embeddings cut loss and boost German, Arabic, Vietnamese","Small PLMs drive big gains for English-centric LLMs via SALT","SALT: linear transfer from PLMs outperforms vocabulary blending"],"cache_read_input_tokens":19456,"weakest_assumption_plain":"The method assumes that a linear map fitted on shared tokens places unseen target-language tokens correctly in the LLM's embedding space; if the PLM and LLM spaces are not linearly related for non-shared tokens, SALT loses its advantage.","fun_headline_variants_meta":{"raw":{"variants":["SALT recycles PLM embeddings for faster, stronger cross-lingual transfer","Per-token linear maps beat embedding swaps in cross-lingual LLM transfer","Recycled PLM embeddings cut loss and boost German, Arabic, Vietnamese","Small PLMs drive big gains for English-centric LLMs via SALT","SALT: linear transfer from PLMs outperforms vocabulary blending"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1237,"prompt_tokens":948,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":190}},"tokens_in":564,"tokens_out":289,"duration_ms":2936,"temperature":1.0,"reasoning_tokens":190,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:59:56.156683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a language pair where the source and target vocabularies share very few tokens, fit SALT on those anchors, and compare the resulting model's downstream accuracy and final training loss to multivariate initialization under identical training; if SALT does not improve, the per-token regression did not generalize beyond its anchors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FOCUS baseline, a weighted-mean embedding initialization from source embeddings that SALT must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pseudo-inverse solution that solves the per-token least-squares transform."}],"review_version":1}