{"id":"0d15d3ba-a79b-46e2-b299-bcc33141c858","arxiv_id":"2412.21140","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Russian-adapted embeddings from foundation LLMs are propagated into instruction-tuned variants, matching original benchmark performance without instruction-tuning data.","lead":"The authors propose Learned Embedding Propagation (LEP), a method that moves Russian-adapted embeddings from a foundation language model into an already instruction-tuned model, skipping the need for Russian instruction-tuning data. Their Darumeru benchmark and experiments on Mistral-7B and LLaMa-3-8B show that LEP variants with added calibration match or slightly beat the original models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Checkpoint and hyperparameter selection on the reporting benchmark (Darumeru) biases LEP's headline results; the claim of competitiveness is vulnerable until a held-out evaluation confirms it.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The reader's weakest_assumption focused on the unvalidated linearity assumption behind Eq. (7), which is a real theoretical gap: the paper assumes W_ru_to_inst ≈ W_inst without an independent check, and the least-squares pseudo-inverse projections in Eqs. (8)-(14) are also unvalidated. However, the most load-bearing concern about the central empirical claim is the benchmark selection procedure. Section 2.4.1 explicitly states that learning rates were chosen based on Darumeru benchmark results, and Figure 2 uses the same benchmark to select checkpoints, with the best checkpoints then reported as final results. Since Darumeru is both the selection criterion and the reported evaluation, the LEP numbers are optimistically biased. This is not merely a minor methodological nit: it directly affects whether the headline comparison to OpenChat 3.5 and LLaMa-3-8B-Instruct is credible. An independent validation split or a separate held-out benchmark would settle this. The linearity assumption should also be checked by comparing an LEP model against a real full-pipeline instruction-tune of the adapted base model, but the selection issue is the more immediate threat to the paper's central claim. Therefore the verdict stays CONDITIONAL, contingent on a held-out evaluation and an independent validation of the propagation formula. The open-source release is a positive factor but does not mitigate the selection bias.","tokens_in":16080,"tokens_out":4961,"duration_ms":53690,"concrete_test":"Re-run the LEP evaluation with an independent validation split: hold out a subset of Darumeru tasks (or use a separate Russian benchmark such as MERA) for learning-rate and checkpoint selection, then report scores only on the remaining held-out tasks. If the best LEP micro-average falls below the OpenChat 3.5 (0.607) or LLaMa-3-8B-Instruct (0.610) baselines, the central claim of competitiveness is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: LEP achieves performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct. The evidence for this claim is compromised by the evaluation protocol. In Section 2.4.1, learning rates are chosen 'based on benchmark results' on Darumeru, and Figure 2 shows benchmark-scored training dynamics with the 'best performing checkpoints' reported in Tables 2-4. Thus Darumeru serves as both the model-selection criterion and the final evaluation instrument. This is selection on the test set: the reported numbers are the maximum over checkpoints and hyperparameters of a noisy evaluation. The baseline models in Table 1 are evaluated once, with no such selection procedure, so the comparison is asymmetrical. The headline result could reflect selection artifacts rather than genuine method quality. This is more load-bearing than the linearity assumption in Eq. (7) (W_ru_to_inst ≈ W_inst), because even if that approximation is loose, LEP could still work empirically; but if the benchmark scores are cherry-picked, the empirical claim itself is unsupported. The paper's open-source release and multiple tokenizer comparisons are genuine strengths, but they do not fix the selection bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Learned Embedding Propagation (LEP), a recipe for adapting 7B-8B open-weight LLMs to Russian while skipping the instruction-tuning step. The pipeline is: (i) train a Russian-optimized tokenizer (BPE, Unigram, Extended, or Optimized), (ii) initialize the new embeddings by averaging overlapping base-token embeddings, (iii) continue-pretrain only the input/output embeddings of the base (foundation) model on Russian text, and (iv) propagate these trained embeddings into the corresponding instruct-tuned model using one of three linear alignment transforms: direct swap, least-squares correction on overlapping tokens (Eqs. 8-11), or a vocabulary-conversion projection (Eqs. 12-14). The derivation is based on a pseudo-linear factorization LLM = I W O (Eq. 3) and the assumption W_ru_to_inst is approximately W_inst (Eq. 7). The authors also introduce Darumeru, a Russian benchmark assembled from MERA, MMLU, RuCoLA, Gazeta, and new copy tasks, and evaluate four vocabulary adaptations of Mistral-7B (with OpenChat 3.5) and LLaMa-3-8B (with its Instruct variant), plus case studies on self-calibration and continued instruction-tuning on Saiga data. Headline results are that LEP with calibration reaches micro-average scores of 0.632 (OpenChat-based) and 0.618 (LLaMa-3-based) versus 0.607 and 0.610 for the original instruct models.","tokens_in":16327,"tokens_out":22062,"duration_ms":208777,"significance":"The manuscript ships all artifacts (models, code, benchmark framework) under open licenses, compares four vocabulary-adaptation strategies systematically on two model families, and introduces a benchmark whose diagnostic tasks (DaruSum abstractive/extractive separation, DaruCopy token-utilization test) address a real gap in Russian LLM evaluation. If the results withstand a clean evaluation, LEP would be a genuinely cheaper alternative to full language adaptation: it needs no Russian instruction data, touches only embedding layers, and reuses existing instruct-tuned weights. The honest Limitations section (need for the foundation model, weak initialization for non-Cyrillic scripts, bounded transfer volume) is to the authors' credit. The concern is that the quantitative claims at the center of the abstract are currently supported only by single-seed runs whose selection protocol is entangled with the reporting benchmark; this is fixable in revision but is not a presentation issue.","major_comments":[{"comment":"The same benchmark, Darumeru, is used both as the model-selection instrument and as the final evaluation. In Sec. 2.4.1 the learning rate is chosen 'based on benchmark results' on Darumeru, and in Sec. 3.2 the tables report 'the best performing checkpoints' from the Darumeru-scored dynamics of Figure 2. The baseline models in Table 1 receive no such selection: each is evaluated once. This asymmetry matters because the claimed advantages are tiny: Table 3 shows OpenChat Extended Conversion at 0.609 versus the original OpenChat at 0.607, and Table 4 shows 'original model + saiga d7' at 0.615 versus LEP Extended + saiga d7 at 0.614, with the +copy-task rows at 0.616 versus 0.618. These margins are within the range that checkpoint and learning-rate selection over a noisy benchmark can produce. The manuscript should separate validation from test: select hyperparameters and checkpoints on one portion of Darumeru (or on another criterion) and report final numbers on an untouched held-out portion or an external benchmark, with the baselines evaluated under the same protocol.","section":"Sec. 2.4.1, Sec. 3.2, Tables 2-4"},{"comment":"All reported scores are single point estimates from single runs: there are no standard deviations, no repeated seeds, and no significance testing, and Darumeru's own run-to-run stability is never characterized. Despite this, the text makes strong comparative claims on very small margins: 'even outperforming the original OpenChat 3.5' for 0.609 versus 0.607 (Table 3), 'by a significant margin' in Sec. 1 for the Figure 1 comparison, and 'new benchmark records' in the Conclusion for differences around 0.01-0.02 (Table 4). Note also that the 0.632 OpenChat-based result does not exceed the 0.639 of Mistral Nemo in Table 1, so 'records' needs qualification. At minimum the paper should report variance (multiple seeds or multiple final checkpoints), state the benchmark's noise level, and temper the superiority language to what single-run differences support.","section":"Sec. 3.2-3.5, Tables 1-4"},{"comment":"The derivation's two structural premises are not validated. First, the factorization LLM = I W O (Eq. 3) treats the entire transformer stack as a single pseudo-linear matrix, and the paper does not test how well this bilinear form approximates the actual models. Second, the step from Eq. (6) to Eq. (7), 'by assuming that the optimal W_ru_to_inst is approximately W_inst', is asserted without evidence or an ablation, although it is the load-bearing premise of the propagation idea. The paper's own results suggest the approximation is lossy: Table 3 shows DaruCopy (RU) falling from 0.972 (LLaMa-3-8B Instruct) to 0.921 under Extended Conversion, and no experiment isolates whether this loss comes from the W mismatch or from the D/U projections. A concrete validation would be to continue-pretrain the instruct-tuned model's own embeddings with frozen inner layers and compare against LEP, or to measure the drift of inner-layer representations after embedding replacement. If the equations are intended only as motivation, the paper should say so and instead justify the method empirically; as written, they are presented as the basis of the three alignment variants in Secs. 2.2.1-2.2.3.","section":"Sec. 2.2, Eqs. (3)-(7)"},{"comment":"The central comparison that would justify the abstract's claim of competitiveness with 'traditional instruction-tuning methods' (running the full language-adaptation pipeline on the same base models and the same vocabulary variants, then instruction-tuning) is not performed. The comparisons in Tables 1-4 are against third-party instruct-tuned models (OpenChat, Saiga, Vikhr, LLaMa-3-8B-Instruct) that were trained with different data and procedures, so the advantage of LEP over the pipeline it is designed to replace is never measured directly. Given the paper's stated purpose of skipping the instruction-tuning step, a within-study control that instruction-tunes a continued-pretrained base model of the same family is needed before the headline claim is fully supported.","section":"Abstract, Sec. 3.5, Tables 3-4"}],"minor_comments":[{"comment":"The citation 'Husain (2024)' in Sec. 1 has no corresponding entry in the reference list; please add it or remove the citation.","section":"Sec. 1, References"},{"comment":"The Conclusion's claim of 'cost-efficient utilization of any instruction-tuned models regardless of openness of their fine-tuning data' overstates the method, since Sec. 5 requires the foundation model to be available; proprietary models without released base weights are out of scope. The phrasing should be tightened.","section":"Sec. 5 vs. Conclusion"},{"comment":"The claims that 'some parts of MMLU and MERA datasets were leaked to training data of Vikhr-5.2 and Qwen-2 7B' and that Vikhr results 'clearly suggest the case of benchmark hacking' are serious allegations about third-party models supported only by a score pattern; they should be presented as hypotheses or accompanied by contamination evidence such as n-gram overlap analysis.","section":"Secs. 3.1 and 4.1"},{"comment":"The description of the Optimized tokenizer is incomplete: 'reducing it to the most common 50% tokens of Russian corpora and then subsequent' is followed by a stray bullet ('Extension to the original size') and an unclear parenthetical. Please describe the construction fully and unambiguously.","section":"Sec. 2.1.1"},{"comment":"Please clarify whether the learning-rate search on 20% of the data was followed by a fresh full-data training run for the final checkpoints, and how the checkpoint-selection points in Figure 2 relate to the training-data split; as written, the reader cannot tell whether the reported models were trained on 20% or on the full corpus.","section":"Secs. 2.4.1 and 3.2"},{"comment":"LLaMa-3-70B-Instruct is used to rephrase MultiQ gold answers, to filter DaruSum examples, and to grade the self-calibration data; the benchmark's and calibration's dependence on LLaMa-3-family judgments should be discussed as a possible source of bias, given that the strongest reported final results are for LLaMa-3-8B-based models.","section":"Secs. 2.3.2-2.3.3, 2.4.2"},{"comment":"Copy-editing is needed: 'don not' in the Abstract, comma-as-decimal in Tables 1-4 versus dot in the text, inconsistent 'LoRA'/'LoRa' in Secs. 2.4.2-2.4.3, unnatural breaks such as 'V ocabulary' in section titles, and several run-on sentences in Sec. 2.3.5.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test assessment that the evaluation protocol (checkpoint and learning-rate selection on the reporting benchmark, single runs) is the more load-bearing problem, and I have kept the W-approximation concern as a separate major because the manuscript frames Eqs. (3)-(7) as a derivation rather than a heuristic. A second editorial concern: the manuscript accuses Vikhr-5.2 and Qwen-2 of benchmark hacking on the basis of score patterns alone; if the journal publishes this, the authors should be asked to soften or substantiate it. Finally, the paper's stated venue is a language/education journal, while the contribution is primarily a systems/engineering result; the Abstract's comparative claims will be read as head-to-head evaluation results, so the validation/test separation should be a condition of acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper has a useful recipe and a new benchmark, but the headline numbers are unreliable because the authors tune on the evaluation set.\n\nThe recipe is LEP, a practical extension of Artetxe et al.'s embedding swap and Ilharco et al.'s task arithmetic to decoder-only instruct-tuned models. The three projection variants are clearly specified, and the tests cover four tokenization strategies on Mistral-7B and LLaMa-3-8B. The authors also built Darumeru, a Russian benchmark with new summarization and copy tasks that diagnose text generation robustness. That is real, useful work, and the code and models are open.\n\nThe problem is the evaluation protocol. In Section 2.4.1, learning rates are chosen based on Darumeru results, and Figure 2 shows benchmark dynamics from which the 'best performing checkpoints' are selected for Tables 2-4. Darumeru thus serves as both the model-selection criterion and the final evaluation instrument. The baseline models in Table 1 are evaluated once with no such selection, so the comparison is asymmetric. The headline claim that LEP is competitive with OpenChat 3.5 and LLaMa-3-8B-Instruct is vulnerable to selection artifacts. This is more load-bearing than the unvalidated linearity assumption W_ru_to_inst ≈ W_inst; that approximation might be loose yet LEP could still work, but if the scores are cherry-picked the empirical claim is unsupported. Also, all scores are single point estimates without seed variance.\n\nThe limitation section is honest about needing both foundation and instruct-tuned versions and about weak adaptation for hieroglyphic languages, which narrows the stated generality.\n\nFor the right reader—someone working on language adaptation or multilingual evaluation—this is worth engaging with. The recipe is concrete, the benchmark is well designed for train-time evaluation, and the open release is a plus. The citation pattern is fine—self-citations are to their earlier tokenization work, which is relevant. But the strong claims need a held-out validation split, multiple seeds, and ideally a full-adaptation baseline. I would send it to peer review, expecting the authors to fix the selection bias and report variance. If they do, this becomes a solid contribution.","headline":"Useful method and benchmark for Russian LLM adaptation, but selection on the evaluation benchmark undermines the headline numbers.","tokens_in":16866,"tokens_out":4864,"would_cite":true,"duration_ms":38879,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes Learned Embedding Propagation, which transfers continued-pretrained Russian embeddings into an instruction-tuned LLM via a linear projection and matches or exceeds the original model without instruction-tuning on…","keywords":["large language models","language adaptation","Russian NLP","embedding propagation","vocabulary extension","continued pretraining","instruction tuning","text generation benchmark"],"falsifier":"Take a base model and its instruct-tuned version, run LEP with the vocabulary-conversion projection, then compare the propagated model's zero-shot Russian benchmark scores against the original instruct-tuned model's scores; if the propagated model does not at least match the original on tasks that do not require new Russian knowledge, the assumption that the transformer body is unchanged is falsified. A more direct check is to measure hidden-layer activation drift on a fixed Russian text set before and after the embedding swap and look for a large drop in cosine similarity.","tokens_in":15811,"feed_emoji":"🇷🇺","tokens_out":9883,"duration_ms":85295,"temperature":0.7,"pith_summary":"The paper proposes Learned Embedding Propagation (LEP), a way to adapt an instruction-tuned large language model to Russian without running instruction-tuning on Russian data. The method continued-pretrains only the input and output embeddings of a base model on Russian text, then propagates those trained embeddings into an existing instruct-tuned version of the same model using one of three linear projections computed from shared tokens. On the authors' Darumeru benchmark, the propagated models match or exceed the original instruction-tuned models on several Russian task groups, and additional calibration steps push them further. If the result holds, language adaptation no longer requires access to a high-quality instruction dataset in the target language, and the instruct-tuned model's proprietary tuning data can be left untouched.","feed_headline":"Embedding swap teaches LLMs Russian without instruction-tuning","feed_subtitle":"A linear correction to swapped embeddings matches or beats the original instruct-tuned models on Russian benchmarks.","key_machinery":"The load-bearing object is the embedding-propagation identity $LLM_{ru/cpt\\to inst} = I_{ru/cpt} D_{inst}^{ru} W_{inst} U_{inst}^{ru} O_{ru/cpt}$, which states that an instruction-tuned variant of a language-adapted model can be obtained by sandwiching the instruction-tuned transformer body between two alignment matrices applied to the continued-pretrained embeddings. Those matrices come from least-squares pseudo-inverses of the overlap between the original and new vocabularies, or from vocabulary-conversion projections. The paper's key claim is that this linear correction is sufficient to skip the instruction-tuning step entirely.","core_discovery":"LEP's central claim is that the entire language-adaptation pipeline can be compressed to two stages: continue-pretraining only the new embedding matrices on target-language text, then transferring those embeddings into any instruct-tuned variant of the same base model through a learned linear transform. The transfer writes the model as $LLM = I W O$, regards the instruct-tuned body as $I_{base} D W_{inst} U O_{base}$, and after continued pretraining approximates $W_{ru\\to inst}\\approx W_{inst}$, yielding the propagated model $I_{ru/cpt} D W_{inst} U O_{ru/cpt}$. The alignment matrices $D$ and $U$ are estimated either as identity (direct swap), as least-squares projections over the common token subset, or as vocabulary-conversion projections. The paper reports that the vocabulary-conversion variant is the most reliable, and that after self-calibration or continued instruction-tuning on a Russian instruction dataset the adapted models reach or surpass the original instruct-tuned quality on the Darumeru benchmark.","pith_inferences":["Editorial inference: the same propagation step should work for any language pair with a reasonable token overlap, not only Russian, because the linear-correction machinery is language-agnostic; the paper only tests Russian.","Editorial inference: if the equal-performance result on OpenChat 3.5 reflects a general property, then instruction-following skill lives mostly in the transformer body, and LEP-style propagation could replace distillation pipelines for adapting proprietary instruct-tuned models to low-resource languages.","Editorial inference: the DaruCopy task, by measuring whether a model can reproduce input text token-for-token, gives a direct probe of output-embedding alignment that could be adopted as a standard regression test for any vocabulary-swapping method.","Editorial inference: a natural next experiment would be applying LEP to a language with near-zero token overlap, such as Japanese or Arabic, where the least-squares projection has little common-token support and the method's graceful-degradation boundary can be mapped."],"forward_implications":["Russian adaptation of 7B-8B models can be done with continued pretraining of embeddings only, plus a projection step, so high-quality Russian instruction data is no longer a prerequisite.","The three propagation variants give a practical recipe: for Mistral-7B, vocabulary extension with conversion projection beats the original OpenChat 3.5; for LLaMa-3-8B-Instruct, no propagation variant fully recovers the original until calibration is applied.","Self-calibration with model-generated answers and copy tasks improves token utilization and summarization, but can hurt comprehension of rare concepts for weaker vocabulary adaptations.","Continued instruction-tuning on the Saiga dataset with an added 2000-example copying task boosts adapted models beyond their non-adapted counterparts on several Russian task groups.","The Darumeru benchmark provides an offline, train-time evaluation with open labels, separating understanding tasks, summarization, and token-copy reliability."],"supporting_citations":[{"why":"Supplies the embedding-swap knowledge-transfer algorithm for encoder models that LEP generalizes to decoder LLMs.","marker":"Artetxe et al., 2019"},{"why":"Provides the result that fine-tuning can be approximated linearly, grounding the assumption that the instruct-tuned body can be treated as fixed.","marker":"Ilharco et al., 2023"},{"why":"Establishes the continued-pretraining language-adaptation pipeline whose instruction-tuning stage LEP eliminates.","marker":"Cui et al., 2023"},{"why":"Supplies the Russian tokenization and embedding-initialization results that motivate the four vocabulary options tested.","marker":"Tikhomirov, 2023"},{"why":"Justifies the subtoken-averaging embedding initialization by concluding it matches task-tailored methods within evaluation variance.","marker":"Tejaswi, 2024"},{"why":"Provides the MERA datasets and Russian evaluation protocol that the new Darumeru benchmark builds on.","marker":"Fenogenova, 2024"},{"why":"Supplies the Gazeta Russian summarization corpus that the DaruSum subset adapts.","marker":"Gusev, 2020"}],"fun_headline_variants":["LEP: Skip instruction-tuning, swap embeddings for Russian LLMs","Learned Embedding Propagation: Fast Russian LLM adaptation","Embedding propagation adapts LLMs to Russian without full retraining","Transfer embeddings, not tuning: LEP for Russian LLMs","Linear embedding swap matches instruct-tuned Russian LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that replacing only the input and output embeddings of an instruction-tuned model with language-adapted embeddings, plus a linear correction estimated on shared tokens, leaves the model's learned reasoning layers intact and functional; if those internal layers shift when the vocabulary changes, the propagated model loses the instruction-tuned behavior.","fun_headline_variants_meta":{"raw":{"variants":["LEP: Skip instruction-tuning, swap embeddings for Russian LLMs","Learned Embedding Propagation: Fast Russian LLM adaptation","Embedding propagation adapts LLMs to Russian without full retraining","Transfer embeddings, not tuning: LEP for Russian LLMs","Linear embedding swap matches instruct-tuned Russian LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3243,"prompt_tokens":1038,"completion_tokens":2205,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":2120}},"tokens_in":654,"tokens_out":2205,"duration_ms":15704,"temperature":1.0,"reasoning_tokens":2120,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:01:37.669310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a base model and its instruct-tuned version, run LEP with the vocabulary-conversion projection, then compare the propagated model's zero-shot Russian benchmark scores against the original instruct-tuned model's scores; if the propagated model does not at least match the original on tasks that do not require new Russian knowledge, the assumption that the transformer body is unchanged is falsified. A more direct check is to measure hidden-layer activation drift on a fixed Russian text set before and after the embedding swap and look for a large drop in cosine similarity.","supporting_citations":[{"cited_title":"Impact of tokenization on llama russian adaptation","cited_arxiv_id":null,"evidence_quote":"Supplies the Russian tokenization and embedding-initialization results that motivate the four vocabulary options tested."},{"cited_title":"Exploring design choices for building language-specific llms","cited_arxiv_id":null,"evidence_quote":"Justifies the subtoken-averaging embedding initialization by concluding it matches task-tailored methods within evaluation variance."},{"cited_title":"Dataset for automatic summarization of russian news","cited_arxiv_id":null,"evidence_quote":"Supplies the Gazeta Russian summarization corpus that the DaruSum subset adapts."}],"review_version":1}