{"id":"36b87669-6ba3-4905-937d-f522885081be","arxiv_id":"2412.06484","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A three-stage continual training recipe (tokenizer change, embedding alignment, full retraining) produces NorMistral-11B, an open Norwegian and Northern Sámi language model that improves on most Norwegian benchmarks and is 30% faster at inference.","lead":"The authors continually train an 11.4-billion-parameter model from Mistral on Norwegian Bokmål, Nynorsk, and Northern Sámi by swapping in a new tokenizer, realigning embeddings, and full retraining. The resulting model, NorMistral-11B, beats prior Norwegian models on most tested tasks and translates better into Northern Sámi, but it loses to its own base model on two benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No same-scale baseline against simple continual training or vocabulary-extension methods exists; Table 4 only compares from-scratch vs. three-stage, so the central quality claim is untested even before the 7B-to-11B transfer question.","rationale":"The reader's weakest_assumption is that the 7B ablations may not transfer to the 11.4B model with a different corpus. That is a valid concern. However, the more fundamental problem is that even the 7B ablations do not test the three-stage recipe against the most relevant baseline: simple continual training that keeps the original tokenizer. Table 4 only compares from-scratch training with the full three-stage warm-start; the difference between those two conditions is initialization, not the tokenizer change or the embedding update. The paper explicitly declines to include simple continual training or vocabulary-extension methods for efficiency reasons, but that decision leaves the quality claim unsupported. A simple continual-training baseline could well match or exceed the three-stage model's quality, since it avoids the initial loss spike and preserves the original embedding space; the paper provides no evidence otherwise. The missing baseline is therefore more load-bearing than the scale-transfer issue: if simple continual training were equally good, the 'novel three-stage' contribution would reduce to an efficiency trick, and the abstract's claim that the approach 'substantially improves downstream performance' would be misleading. I still agree with the reader's CONDITIONAL verdict because the paper delivers a useful open model, a new Sámi corpus, and reproducible code; the efficiency claim is solid. The condition should explicitly require the missing baselines, at least at 7B scale, before the methodological claim is accepted.","tokens_in":38633,"tokens_out":6468,"duration_ms":61330,"concrete_test":"Train a 7B model with the same base model, the same Norwegian Colossal Corpus, and the original Mistral tokenizer, for the same number of steps and hyperparameters as the three-stage 7B model in Table 4, then evaluate on the same four benchmarks. If this simple continual-training model matches or exceeds the three-stage model's scores (NorQuAD 64.8, NoReC 84.9, NRK 57.9, Tatoeba 57.2), the claim that tokenizer replacement and embedding update improve quality is not supported. This test isolates the recipe from the warm-starting effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the three-stage continual training recipe (new tokenizer, embedding update, full training) substantially improves downstream performance while increasing inference efficiency. The efficiency part is well supported by Table 1 and Appendix A. The quality part, however, is not established by the experiments as designed. In Section 5 ('From scratch vs. warm-starting'), the only continual-training comparison is between a from-scratch 7B model and a 7B model warm-started via the full three-stage recipe, both trained on the Norwegian Colossal Corpus with the same new tokenizer. This comparison isolates warm-starting, not the three-stage recipe. The paper explicitly states that simple continual training and adapter tuning are not considered 'because they necessarily lead to inefficient inference' – but that rationale concerns efficiency, not quality. To support the claim that the three-stage method improves performance, one must show it does not sacrifice quality relative to simple continual training on the same base model and corpus. No such baseline appears anywhere. Similarly, the contribution of Stage 2 (embedding update) is asserted to prevent loss spikes and catastrophic forgetting, but no ablation skips it. Consequently, even at 7B scale on the same corpus, the paper does not establish that the three-stage recipe is superior in quality to existing vocabulary-adaptation or simple continual-training methods. The released 11.4B model adds further confounds: a different, larger corpus mixture and a different scale, with no same-scale continual-training baseline. The claim that the recipe 'substantially improves' performance is therefore load-bearing on an untested assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage continual pretraining recipe for adapting a 12B English-centric model (Mistral-Nemo-12B) to Norwegian Bokmål, Nynorsk, and Northern Sámi: (1) train a new domain-optimized BPE tokenizer, (2) realign input/output embeddings via 1,000 steps of frozen-backbone training, and (3) train the full model. The authors release NorMistral-11B, three 7B checkpoints, training/evaluation code, and a new Northern Sámi web corpus. They report that the new tokenizer cuts sequence lengths by roughly 30% and gives a corresponding inference speedup (Table 1, Appendix A), and that NorMistral-11B outperforms several existing Norwegian models on most of the NorEval benchmarks (Table 2). The paper also describes a hybrid masked-causal training objective and demonstrates flexible use of the model as a causal, prefix, or bidirectional LM (Section 3.2, Table 3). Section 5 presents ablations at 7B scale, including architecture choice, from-scratch vs. warm-starting, the hybrid objective, and number of training steps. The central claim is that this three-stage continual training substantially improves both downstream performance and inference efficiency for low-resource languages.","tokens_in":38881,"tokens_out":4919,"duration_ms":51906,"significance":"If the central claim were fully supported, this would be a practically valuable recipe for adapting large English-centric models to lower-resource languages, because it reduces inference cost while reusing existing pretrained knowledge. The paper has clear strengths: the models, code, evaluation prompts, and the new Sámi corpus are openly released; the tokenizer efficiency evidence in Table 1 and Appendix A is concrete and machine-checkable; and the hybrid masked-causal objective is an interesting design choice whose downstream effects are partially documented. The main weakness is that the performance advantage of the three-stage recipe over simple continual training is not established, because no same-scale baseline of that kind is run. The 7B ablations are on a different corpus and scale than the released model, and the evaluation protocol reports maximum scores across prompts, which can inflate apparent gains. These issues are load-bearing for the paper's claimed contribution.","major_comments":[{"comment":"The only continual-training comparison is 'init. from scratch' versus 'three-stage continual' at 7B scale on the Norwegian Colossal Corpus. The paper explicitly does not consider simple continual training or adapter tuning because they 'necessarily lead to inefficient inference' (Section 5). That rationale addresses inference cost, not model quality. The abstract and introduction claim that the three-stage approach 'substantially improves the downstream performance'; to support that claim, the paper needs a baseline that continually trains the same base model on the same corpus while keeping the original tokenizer (or extending the vocabulary) and evaluates at the same scale. Without such a baseline, the evidence does not show that the three-stage recipe is better than simpler alternatives in quality, only that it is faster. This is a central, load-bearing omission.","section":"Section 5, Table 4"},{"comment":"The embedding-update stage is asserted to prevent loss spikes and catastrophic forgetting, but no ablation removes or alters this stage. Since the 'three-stage' method is the paper's main contribution, the contribution of Stage 2 should be measured directly, for example by comparing full training from randomly initialized new embeddings, from averaged sub-token embeddings without realignment, and from the proposed 1,000-step realignment. As written, the necessity of the second stage is an unverified component of the central recipe.","section":"Section 3.1, Stage 2"},{"comment":"NorMistral-11B regresses relative to Mistral-Nemo-12B on Belebele (56.7 vs. 62.8) and NorOpenBookQA (77.9 vs. 86.9 on Bokmål, 77.8 vs. 86.7 on Nynorsk). The text acknowledges this and says it 'requires a further study,' but the abstract and introduction still claim that the approach 'substantially improves the downstream performance' without qualifying these exceptions. Given that the base model outperforms the adapted model on two of the benchmarks, the unconditional performance claim should be revised to state which tasks improve and which regress, and ideally the paper should provide some analysis of why these regressions occur rather than leaving them unexplained.","section":"Section 4, Table 2"},{"comment":"The evaluation protocol reports the maximum score across five prompt templates and across chosen k-shot settings. The full per-prompt results in Appendix B show very high prompt sensitivity (e.g., Belebele scores for NorMistral-11B range from 22.8 to 56.7 in Table 6). Reporting the maximum rather than a mean or median can systematically inflate performance and makes the headline comparisons sensitive to prompt-search luck. The paper should either justify this protocol, report both the maximum and the mean/median with variance, or provide the full distribution of scores in the main tables so that the SOTA claim is robust to prompt selection.","section":"Section 3.3 and Table 2 caption"}],"minor_comments":[{"comment":"The text refers to 'NorOpenBookQ' but the benchmark is named NorOpenBookQA; this typo should be corrected.","section":"Section 4, first paragraph"},{"comment":"The phrase 'ranging fron 15M parameters' should read 'ranging from 15M parameters'.","section":"Section 6, 'Norwegian language models'"},{"comment":"The phrase 'in-context-lerning' should be 'in-context learning'.","section":"Appendix A, first sentence"},{"comment":"The text says 'The complete evaluation on ASK-GEC is provided in Table 15,' but the table immediately following is Table 14; the citation is off by one.","section":"Appendix B.8"},{"comment":"The sentence 'We used the following four prompt templates for testing all language models on translation to Northern Sámi' appears under the English-to-Bokmål translation subsection; it should refer to Bokmål translation.","section":"Appendix B.10.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is an engineering contribution with a valuable open release. The main concern is not novelty or circularity but experimental design: the paper's central performance claim is not tested against the most natural baseline, simple continual training at the same scale. The evaluation protocol's use of maximum scores across prompts also makes the headline numbers fragile. If the authors add the missing baseline or substantially qualify the claim, the paper could become acceptable. I would suggest that the editor ask the authors to include a same-scale simple-continual-training comparison in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful paper with a real artifact, and the inference-efficiency story is solid. But the headline claim that the three-stage continual-training recipe improves quality is not actually tested. The ablations in Table 4 compare a from-scratch 7B model with a 7B model trained via the full three-stage recipe on the same corpus. That isolates warm-starting, not the recipe. There is no comparison to simple continual training on the same base model, nor to the vocabulary-extension methods the paper cites (de Vries and Nissim; Marchisio et al.). The authors say they skip those because they 'necessarily lead to inefficient inference' – which is an efficiency argument, not a quality argument. So the central quality claim rests on an untested assumption.\n\nWhat the paper does well: NorMistral-11B is openly released, with training and evaluation code, and a new Northern Sámi web corpus. The tokenizer replacement clearly delivers: ~30% shorter sequences and measurable inference speedups (Appendix A), with 800M fewer parameters. The 7B ablation does show warm-starting beats from-scratch by a wide margin, which is a useful result on its own. The limitations section is honest: they flag the Belebele and NorOpenBookQA regressions, and they explicitly say stronger Sámi claims would need a more robust benchmark.\n\nThe soft spots, in proportion: (1) Reporting the maximum over five prompts and three shot counts is optimistic; a mean or a fixed prompt would be easier to interpret. (2) The Sámi evaluation is 53 translation examples; the paper itself admits this, but the abstract still says 'strong capabilities in Northern Sámi,' which overstates it. (3) The regressions on Belebele and NorOpenBookQA are acknowledged but not analyzed; that matters because those are exactly the benchmarks where the base model was strong. (4) The hybrid masked-causal objective shows no clear gain in Table 4; the paper waves this away, which is fine, but then it shouldn't be listed as a contribution with 'demonstrated benefits.'\n\nWho this is for: practitioners building LLMs for mid- and low-resource languages, and people working on vocabulary adaptation. They will get value from the artifacts and the efficiency measurement. The methodological claim about the three-stage recipe needs more work before I'd trust it.\n\nRecommendation: send it to peer review, but with a clear request to add same-scale baselines (simple continual training, a cited vocabulary-extension method) and to either average over prompts or pre-register the choice. The artifacts and the efficiency result justify the review; the current evidence does not justify the strong quality claim.","headline":"Worth reading for the open model and the efficiency result, but the paper's central quality claim for the three-stage recipe is not supported by the experiments as run.","tokens_in":39591,"tokens_out":2635,"would_cite":true,"duration_ms":23700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage continual training recipe adapts a 12B English model to Norwegian and Sámi, beating open baselines on most native tasks while running 30% faster.","keywords":["continual pretraining","tokenizer adaptation","low-resource languages","Norwegian","Northern Sámi","language model","embedding realignment","hybrid masked-causal"],"falsifier":"Train an 11.4B model from scratch on the same 250B-token mixture, or train an 11.4B model with tokenizer replacement but without the embedding-update stage, and compare on NorQuAD, NoReC, and Tatoeba; if either matches NorMistral-11B, the three-stage claim is not load-bearing.","tokens_in":38307,"feed_emoji":"🇳🇴","tokens_out":4100,"duration_ms":38049,"temperature":0.7,"pith_summary":"The paper asks whether a large English-centric model can be made to serve languages with far less data, such as Norwegian and especially Northern Sámi. It proposes a three-stage continual training method: build a byte-level BPE tokenizer for the target corpus, realign the embedding matrix to the new vocabulary while freezing all other parameters, then train the full model. The resulting NorMistral-11B outperforms comparable open models on most Norwegian benchmarks and is more than 30% faster on Norwegian inputs because its tokenizer cuts average sequence length by about 30%. If this recipe holds, it gives low-resource languages a practical path to large models without training from scratch.","feed_headline":"Three-stage retraining turns a 12B English model into Norway's LLM","feed_subtitle":"New tokenizer, realigned embeddings, and full training beat open baselines on Norwegian tasks at 30% lower inference cost.","key_machinery":"The three-stage continual pretraining pipeline: Stage 1 trains a new byte-level BPE tokenizer on the target corpus; Stage 2 copies and averages original embeddings into the new embedding matrix and trains only embeddings for 1,000 steps to avoid loss spikes and catastrophic forgetting; Stage 3 unfreezes all parameters and trains the full model on 250B tokens. A secondary mechanism is the hybrid masked-causal objective, which combines standard causal language modeling with masked next-token prediction so the model can be used in multiple inference modes.","core_discovery":"The central claim is that replacing the tokenizer, realigning embeddings, and then training the whole model efficiently transfers knowledge from an English-centric 12B model to Bokmål, Nynorsk, and Northern Sámi. The paper reports that NorMistral-11B reaches state-of-the-art results on most tasks in the NorEval suite, improves English-to-Northern Sámi translation from near zero to about 50 BLEU at 16 shots, and runs more than 30% faster than Mistral-Nemo-12B because its 51,200-token vocabulary produces shorter sequences. The authors also show that the same model can be used as a causal generator, a masked language model, or a prefix language model by mixing causal language modeling with masked next-token prediction at a 90/10 ratio.","pith_inferences":["The recipe is likely to transfer to other small languages with closely related large languages, because the corpus already mixes Bokmål with Danish, Swedish, Icelandic, Faroese, English, and code.","The 7B ablations that justify the three-stage recipe may not fully determine the 11.4B result; the released model uses a different corpus mixture, so the relative value of the embedding-update stage at scale remains unproven.","A fairer test of the method would compare against simple vocabulary extension plus full training under the same inference budget, since the paper excludes such baselines on efficiency grounds."],"forward_implications":["A language with only tens of millions of words can be added to an existing large model by continued training on upsampled data, with no architecture change.","Replacing the tokenizer yields a substantial inference speedup: 30% shorter sequences translate to more than 30% faster per-sample processing.","The same checkpoint can serve as a generative model and as a bidirectional encoder, so a single training run covers both use cases.","Continual training can push performance past the base model on native-language benchmarks while risking some regression on multilingual datasets like Belebele."],"supporting_citations":[{"why":"Supplies the embedding-alignment method used in Stage 2 of the continual training recipe.","marker":"(de Vries and Nissim, 2021)"},{"why":"Provides the data-constrained scaling laws that justify repeating target-language data up to 16 times without major negative effects.","marker":"(Muennighoff et al., 2023)"},{"why":"Provides the Chinchilla scaling laws used to argue that 250B tokens are compute-optimal for an 11.4B model.","marker":"(Hoffmann et al., 2022)"},{"why":"Supplies the Norwegian Colossal Corpus used for the methodological comparisons and part of the final training mixture.","marker":"(Kummervold et al., 2022)"},{"why":"Supports the hybrid masked-causal training objective by showing benefits of combining causal and masked next-token prediction.","marker":"(Charpentier and Samuel, 2024)"},{"why":"Defines the Mistral/Llama-style transformer architecture inherited and continually trained by NorMistral.","marker":"(Jiang et al., 2023)"}],"fun_headline_variants":["Retrained 12B model achieves SOTA on Norwegian and Sámi with 30% faster inference","Three-stage retraining turns English 12B into Norwegian 11B SOTA","New tokenizer and full retrain yield faster Norwegian LLM for Bokmål, Nynorsk, Sámi","NorMistral: 11B model from continual training, cutting inference cost by 30%","English to Norwegian: how a tokenizer swap and retraining built a better LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evidence that the three-stage recipe beats simpler continual training comes from 7-billion-parameter models trained on one corpus, while the released model is 11.4 billion parameters trained on a different, larger mixture, so the paper assumes the 7B ranking carries over to the final model and to the tiny Northern Sámi portion.","fun_headline_variants_meta":{"raw":{"variants":["Retrained 12B model achieves SOTA on Norwegian and Sámi with 30% faster inference","Three-stage retraining turns English 12B into Norwegian 11B SOTA","New tokenizer and full retrain yield faster Norwegian LLM for Bokmål, Nynorsk, Sámi","NorMistral: 11B model from continual training, cutting inference cost by 30%","English to Norwegian: how a tokenizer swap and retraining built a better LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":2038,"prompt_tokens":810,"completion_tokens":1228,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":1105}},"tokens_in":426,"tokens_out":1228,"duration_ms":11911,"temperature":1.0,"reasoning_tokens":1105,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:36:35.589575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an 11.4B model from scratch on the same 250B-token mixture, or train an 11.4B model with tokenizer replacement but without the embedding-update stage, and compare on NorQuAD, NoReC, and Tatoeba; if either matches NorMistral-11B, the three-stage claim is not load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Chinchilla scaling laws used to argue that 250B tokens are compute-optimal for an 11.4B model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Norwegian Colossal Corpus used for the methodological comparisons and part of the final training mixture."},{"cited_title":"GPT or BERT: why not both?","cited_arxiv_id":"2410.24159","evidence_quote":"Supports the hybrid masked-causal training objective by showing benefits of combining causal and masked next-token prediction."}],"review_version":1}