{"id":"ce2c15aa-a304-4c99-82fb-af8566876eb4","arxiv_id":"2412.07633","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Continued pretraining with LoRA and a Dutch-specific tokenizer improves Llama-2's Dutch, but gives limited gains for already-multilingual Llama-3.","lead":"This paper tried to teach English-focused Llama AI models Dutch by feeding them 32 billion tokens of Dutch text and tweaking the tokenizer. It found the tricks helped for the older Llama-2, but newer Llama-3 already spoke Dutch well enough that extra training barely mattered.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Llama-3 conclusion that continued pretraining adds little is confounded: Llama-3-ChocoLlama-instruct differs from Meta-Llama-3-Instruct in both continued pretraining and posttraining recipe, so the comparison cannot isolate the effect of continued pretraining.","rationale":"The reader's weakest_assumption concerns evaluation reliability, specifically whether ChocoLlama-Bench and GPT-4o judge scores validly measure Dutch generation quality. That is a legitimate concern, and it is part of why the paper's Llama-3 conclusion is fragile: the standard benchmarks in Table 2 actually favor Llama-3-ChocoLlama-8B-instruct over Meta-Llama-3-8B-Instruct, while ChocoLlama-Bench favors Meta's model, and the paper resolves this conflict by asserting the new benchmark is more reliable without validation. However, the more load-bearing issue is internal to the experimental design: even if ChocoLlama-Bench were perfectly reliable, the comparison between Llama-3-ChocoLlama-8B-instruct and Meta-Llama-3-8B-Instruct does not isolate continued pretraining because the two models also use different posttraining recipes. The authors themselves state that the posttraining applied to Meta-Llama-3-8B-Instruct is likely significantly better than the one they adopt. Consequently, the abstract's claim that continued pretraining yields limited gains for Llama-3 while posttraining matters more is not supported by the current evidence. A same-posttraining ablation, or at least a version of the ChocoLlama posttraining applied with and without Dutch continued pretraining, would settle the question. I do not believe this requires a different verdict than the reader's CONDITIONAL: the paper remains a useful empirical contribution on LoRA-based adaptation and tokenizer reinitialization for Llama-2, and the problematic comparison can be fixed by an additional experiment. The reader's verdict already conditions acceptance on addressing evaluation issues; this confound should be added to those conditions.","tokens_in":17555,"tokens_out":4799,"duration_ms":40831,"concrete_test":"Run a 2x2 ablation on Llama-3-8B with the same posttraining pipeline used in the paper (Vanroy SFT + Zephyr DPO): (A) Meta-Llama-3-8B base alone; (B) Meta-Llama-3-8B base + Dutch continued pretraining (ChocoLlama-base); then apply the identical SFT/DPO to both, yielding (A') and (B'). Compare (A') vs (B') on ChocoLlama-Bench and the four Table 2 benchmarks. If (B') is not better than (A'), the continued-pretraining-is-limited conclusion holds; if (B') is better, the paper's comparison was confounded and its central claim reverses. Also report human agreement on a sample of ChocoLlama-Bench judgments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2.4 and Section 6 draw the central lesson that for multilingual foundation models 'language adaptation techniques may benefit more from focusing on language-specific posttraining rather than on continued pretraining.' This rests on comparing Llama-3-ChocoLlama-8B-instruct (Meta-Llama-3-8B base + Dutch continued pretraining + SFT/DPO from Vanroy/Zephyr) against Meta-Llama-3-8B-Instruct (base + Meta's own instruction tuning). These models differ in two variables at once: whether Dutch continued pretraining was performed and which posttraining recipe was used. The paper itself concedes in Section 5.2.4 that Meta's instruction tuning is likely better than the adopted recipe. Therefore the observed lower ChocoLlama-Bench score cannot be attributed to continued pretraining being unhelpful; it may simply reflect an inferior posttraining stage. The discrepancy with Table 2, where Llama-3-ChocoLlama-8B-instruct actually outperforms Meta-Llama-3-8B-Instruct on average, makes the interpretational risk concrete. The paper tries to resolve the conflict by asserting ChocoLlama-Bench is 'more reliable,' but provides no validation. Without a same-posttraining ablation, the headline claim about posttraining over continued pretraining is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts Llama-2 and Llama-3 to Dutch through LoRA-based continued pretraining on 104 GB of Dutch text (32B Llama-2 tokens), optionally replacing the tokenizer with a Dutch-specific BPE tokenizer plus embedding reinitialization, and then applying SFT and DPO from prior work. The authors evaluate the resulting ChocoLlama models on Dutch translations of ARC, HellaSwag, MMLU, and TruthfulQA, and on a new benchmark (ChocoLlama-Bench) in which GPT-4o judges side-by-side responses to 56 Dutch prompts. They report that LoRA-based continued pretraining improves Llama-2's Dutch performance, that the Dutch-tokenizer variant outperforms the original-tokenizer variant, and that the same recipe yields limited gains for Llama-3, leading to the conclusion that language adaptation for multilingual foundation models should focus on posttraining rather than continued pretraining.","tokens_in":17896,"tokens_out":5879,"duration_ms":50007,"significance":"The work is practically valuable: it releases models, code, and a new Dutch benchmark, and it applies trans-tokenization with embedding reinitialization to autoregressive models at a scale (32B tokens) not previously demonstrated. If the claims hold, the paper would be a useful case study for low-resource language adaptation with parameter-efficient methods. The main significance, however, hinges on the reliability of the evaluation. The paper's key qualitative conclusions—especially the claim that continued pretraining yields limited gains for Llama-3—are drawn from an unvalidated LLM-as-a-judge benchmark, and the quantitative deltas in the standard benchmarks are small and lack statistical support. These issues prevent the headline lessons from being established as stated, though they are addressable with additional experiments and analysis.","major_comments":[{"comment":"The paper asserts that ChocoLlama-Bench 'offer[s] a more comprehensive and reliable evaluation' (Section 6) and that this reliability is 'empirically' found, but no evidence is provided: there is no human evaluation, no inter-annotator agreement, no comparison with the quantitative benchmarks, and no analysis of GPT-4o judge bias. This matters because the benchmark is used to overturn the quantitative result in Table 2, where Llama-3-ChocoLlama-8B-instruct outperforms Meta-Llama-3-8B-Instruct on average (0.53 vs 0.51). The central conclusion that continued pretraining adds little for Llama-3 depends critically on this unvalidated judge, so the reliability claim is load-bearing and must be supported.","section":"Section 6 and Section 5.2.1"},{"comment":"The comparison underlying the headline lesson 'posttraining rather than continued pretraining' is confounded. Llama-3-ChocoLlama-8B-instruct differs from Meta-Llama-3-8B-Instruct in two variables: whether Dutch continued pretraining was performed and which posttraining recipe was used (Vanroy/Zephyr SFT+DPO vs Meta's own instruction tuning). The paper itself acknowledges (Section 5.2.4) that Meta's instruction tuning is likely better than the adopted recipe, so the observed ChocoLlama-Bench outcome cannot isolate the effect of continued pretraining. A same-posttraining ablation, such as applying the same Dutch posttraining to both Meta-Llama-3-8B and Meta-Llama-3-8B-Instruct, or applying Meta's posttraining to Llama-3-ChocoLlama-base, is needed before drawing the stated conclusion.","section":"Section 5.2.4"},{"comment":"The comparison between the original-tokenizer model (ChocoLlama-2-7B-base) and the Dutch-tokenizer model (ChocoLlama-2-7B-tokentrans-base) is confounded by training duration and token count: Table 1 shows the tokentrans model was trained for 43.8k steps on 22.6B tokens, whereas the original-tokenizer model was trained for 62.1k steps on 32B tokens. Because both are trained for exactly one epoch, the new tokenizer changes both the representation and the number of gradient updates. To attribute the observed improvement (e.g., 0.45 vs 0.41 average in Table 2) to tokenizer modification with weight reinitialization, one would need to control for steps or tokens, for example by training the original-tokenizer model for the same number of steps or the trans-tokenized model for the same number of tokens.","section":"Section 2.2, Table 1, and Table 2"},{"comment":"The quantitative gains are small and no measures of uncertainty or significance are reported: the best ChocoLlama-2-instruct average is 0.46 versus 0.43 for the original-tokenizer variant, and the improvement over Llama-2-7b-chat is 0.05; the per-benchmark differences are often within a few points (e.g., HellaSwag 0.62 vs 0.57, MMLU 0.34 vs 0.33). Similarly, the head-to-head result in Table 3 (24 wins vs 15 wins, with 17 ties, out of 56 turns) is not statistically significant under a simple binomial test on non-tied turns. Without confidence intervals, standard errors, or a significance test, the claims that tokenizer modification 'improves' performance and that LoRA 'effectively scales' rest on effects that could be noise. At minimum, the authors should report variance across benchmark items or repeated evaluations, or temper the strength of the claims.","section":"Section 5.1 and Table 3"}],"minor_comments":[{"comment":"The notation in Equation (1) is inconsistent: the formula uses Pθ(Dbatch) and |Tθ(Dref)|/|Tθref(Dref)|, but the surrounding text says Pθ is the perplexity evaluated on Dref. Please clarify which dataset is used in the numerator and why the batch index appears in one place and Dref in another.","section":"Equation (1)"},{"comment":"The quantitative benchmarks use GPT-3.5-turbo Dutch translations of ARC, HellaSwag, MMLU, and TruthfulQA, but the paper does not discuss or validate the quality of these translations. A brief comment on translation fidelity, or a reference to prior validation, would help the reader judge whether the multiple-choice scores are meaningful for Dutch.","section":"Section 5.1"},{"comment":"The two comparisons against Llama-2-7b-chat give seemingly inconsistent strengths: ChocoLlama-2-7B-instruct wins 78% of turns, while ChocoLlama-2-7B-tokentrans-instruct wins only 66%, yet the tokentrans model beats the nontokentrans model in the direct comparison. This is possible because of ties, but the paper does not discuss the pattern; reporting only win percentages can be misleading when tie rates differ.","section":"Table 3"},{"comment":"References [46] and [47] are the same paper (Zheng et al., MT-Bench / Chatbot Arena); one should be removed or both should be merged.","section":"References"},{"comment":"The description of ChocoLlama-Bench prompts as '30 Dutch prompts and follow-up prompts' is ambiguous; it would be clearer to say 30 multi-turn conversations comprising 56 turns total.","section":"Section 5.2.1"},{"comment":"The phrase 'we empirically find this benchmark to offer a more comprehensive and reliable evaluation' is not supported by any empirical comparison; consider replacing 'empirically' with a direct description of what was measured, or softening the claim until supporting evidence is added.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about many of its limitations and the open-sourced models and benchmark are a useful contribution. The main concern is that the central comparative claims—particularly the Llama-3 'posttraining over continued pretraining' lesson—are not yet established because of the confounded comparison and the unvalidated evaluation benchmark. The authors can likely address this with a same-posttraining ablation, human evaluation of ChocoLlama-Bench, and significance testing. I would not reject the paper, but the revision needs to be substantive rather than purely cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on ChocoLlama. It is a genuine contribution: as far as I know, the first application of tokenizer translation to an autoregressive Llama-2 with embedding reinitialization, a 32B-token LoRA continued pretraining run, and a new open benchmark, ChocoLlama-Bench. Models, code and benchmark are released. That is real value for anyone working on lower-resource language adaptation.\n\nThe Llama-2 results are credible, though modest: the token-translated instruct model averages 0.46 on the four Dutch multiple-choice benchmarks versus 0.43 for the original-tokenizer variant, and leads on ChocoLlama-Bench 43%-27% with 30% ties. No confidence intervals or significance tests, so treat the deltas as suggestive rather than established. Also, because the translated tokenizer has a larger vocabulary and more trainable parameters, part of the gain could be capacity rather than tokenization per se.\n\nThe bigger problem is the Llama-3 conclusion. The paper argues that continued pretraining adds little and that posttraining matters more, comparing Llama-3-ChocoLlama-instruct (base + CPT + Vanroy SFT/DPO) against Meta-Llama-3-Instruct (Meta's posttraining). Two variables change at once, and the authors themselves concede in Section 5.2.4 that Meta's instruction tuning is likely significantly better than the recipe they adopted. So the observed lower ChocoLlama-Bench score cannot be attributed to continued pretraining being unhelpful. The discrepancy with Table 2, where the ChocoLlama model does better on the quantitative benchmarks, makes the danger concrete. They try to resolve it by asserting ChocoLlama-Bench is more reliable, but no validation is given. That assertion is not enough.\n\nWhat the paper does well is to document the recipe, the data mix, the perplexity trends, and the honest discussion of limitations. It also correctly notes the benchmark dependency on GPT-4o's Dutch proficiency. So my advice: the Llama-3-specific claim should be reframed as a hypothesis, and the ChocoLlama-Bench reliability statement needs an external check (e.g., human ratings or a second judge) before it is used to override a quantitative benchmark. The Llama-2 story can stand with some hedging.\n\nWho is this for? Anyone adapting LLMs to a mid-resource language with PEFT; it gives a concrete, reproducible protocol and a new evaluation resource. It deserves a serious referee; it is a solid empirical paper with a fixable overreach in interpretation.","headline":"Genuine empirical contribution with a solid Llama-2 recipe, but the Llama-3/posttraining conclusion is confounded and the self-built benchmark's reliability claim is unvalidated.","tokens_in":18414,"tokens_out":4773,"would_cite":true,"duration_ms":35894,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that continued pretraining with LoRA at 32B tokens can adapt Llama-2 to Dutch, that a Dutch-specific tokenizer with embedding reinitialization improves quality and efficiency, and that the same recipe adds little to the…","keywords":["language adaptation","Dutch","LoRA","tokenizer translation","embedding reinitialization","continued pretraining","multilingual LLM evaluation","ChocoLlama-Bench"],"falsifier":"Run the same 56-prompt side-by-side comparisons with human raters or a different judge model; if human preferences do not match the LLM-as-a-judge's preferences for the tokenizer-translated versus original-tokenizer ChocoLlama models, the claim that tokenizer modification improves language generation collapses. Alternatively, inspect the Dutch benchmark translations for translation errors that change the correct answer.","tokens_in":17415,"feed_emoji":"🦙","tokens_out":6666,"duration_ms":50969,"temperature":0.7,"pith_summary":"The paper reports an attempt to adapt Llama-2 and Llama-3, two English-centric large language models, to Dutch. The central claim is that continued pretraining with Low-Rank Adaptation (LoRA), updating only about 8-12% of parameters during pretraining on 32 billion Dutch tokens, can successfully teach Llama-2 Dutch. A second claim is that replacing Llama-2's tokenizer with a Dutch-trained one, plus reinitializing embeddings to match old tokens, improves quality and reduces token count by 29.4%. The paper also argues that the same continued-pretraining recipe gives only limited gains for Llama-3, which already has strong Dutch abilities, concluding that for increasingly multilingual foundation models, language-specific posttraining may matter more than continued pretraining. These findings matter because they suggest a cost-effective recipe for adapting existing models to lower-resource languages, and they refine expectations about when further pretraining is worth the effort.","feed_headline":"LoRA teaches Llama-2 Dutch, but Llama-3 gains little","feed_subtitle":"A Dutch tokenizer with embedding reinitialization beats the original; newer models need posttraining, not more pretraining.","key_machinery":"The two load-bearing mechanisms are (1) LoRA applied at scale, training low-rank updates on all target modules (r=8, alpha=32) with full updates to the embedding and output head, giving 544M-1.07B trainable parameters (7.75-11.77% of each model); and (2) tokenizer translation with embedding reinitialization, in which a new Dutch BPE tokenizer (50k tokens) replaces the original and each new token's embedding is initialized from a semantically similar old token before continued pretraining. The paper also introduces a normalized-perplexity formula (Equation 1) that rescales token-level perplexity to a common reference tokenizer, allowing fair comparison of training curves across different tokenization schemes.","core_discovery":"On the paper's own terms, the discovery is twofold. First, a parameter-efficient continued-pretraining scheme, in which LoRA adapters are trained on all attention and MLP layers while the embedding and head are fully updated, is sufficient to make Llama-2 respond in fluent Dutch: the adapted models beat the original Llama-2 on both standard multiple-choice benchmarks and the new ChocoLlama-Bench conversation benchmark. Second, translating the tokenizer, replacing the English-trained vocabulary with a Dutch-trained BPE vocabulary of 50,000 tokens and initializing each new embedding from a semantically similar old token, yields a further gain: the tokentrans model beats the original-tokenizer model, particularly on writing tasks, and compresses Dutch text into 29.4% fewer tokens. When the same recipe is applied to Llama-3, which was released mid-project and already outperformed the adapted Llama-2, the continued pretraining produces a model that wins on multiple-choice benchmarks but loses on ChocoLlama-Bench against the original Llama-3-instruct; the paper reads this as evidence that continued pretraining yields diminishing returns for models with strong multilingual capabilities.","pith_inferences":["If the LLM-as-a-judge scores on ChocoLlama-Bench are even approximately right, the tokenizer-translation result has a strong practical corollary: adapting the tokenizer should be considered for any language whose tokenization differs substantially from English, and the same embedding-reinitialization trick could be combined with other parameter-efficient methods.","The Llama-3 result suggests a testable hypothesis the paper leaves implicit: continued pretraining on a language already well represented in the base model may be near-neutral or even harmful for conversational quality, and posttraining on high-quality native data is the higher-leverage intervention.","The 29.4% token reduction implies an inference-cost benefit that the paper does not quantify; one could measure tokens per Dutch sentence for the two ChocoLlama-2 variants to estimate speed and memory savings in practice.","Because the posttraining data used in the paper cannot be used commercially, the net practical value of the recipe depends on retraining the SFT and DPO stages with permissive data; a reader could test whether the base-model gains survive a fully open posttraining pipeline."],"forward_implications":["Continued pretraining with LoRA at the 32B-token scale is a viable, low-cost route for adapting primarily English models to a new language: about 10% of parameters are trained, and the resulting models use Dutch consistently.","A language-specific tokenizer with embedding reinitialization not only improves Dutch generation quality for Llama-2 but cuts token count by 29.4%, reducing training and inference cost.","For models like Llama-3 that already have broad multilingual coverage, further Dutch pretraining gives at best marginal gains on standard benchmarks and a small regression on conversational quality, so the bottleneck shifts to posttraining.","The open ChocoLlama-Bench (56 Dutch prompts across reasoning, comprehension, and writing, scored by an LLM judge) provides a new resource for comparing Dutch generation quality beyond multiple-choice accuracy.","The qualitative finding that the original Llama-2 often answers in English while the adapted models answer in Dutch suggests the adaptation genuinely changes the model's language behavior, not just benchmark scores."],"supporting_citations":[{"why":"Defines LoRA, the parameter-efficient method the paper scales up for continued pretraining on 32B tokens.","marker":"[20]"},{"why":"Introduces the embedding-reinitialization strategy that lets a Dutch-trained tokenizer replace the original without catastrophic forgetting.","marker":"[32]"},{"why":"Specifies Llama-2, the English-centric base model whose Dutch adaptation the paper demonstrates and whose tokenizer counts the 32B-token corpus.","marker":"[39]"},{"why":"Specifies Llama-3, the newer multilingual base model whose strong Dutch capabilities set the comparison that limits the continued-pretraining claim.","marker":"[2]"},{"why":"Provides OSCAR, the source of 93GB of the Dutch pretraining data and the corpus used to train the Dutch BPE tokenizer.","marker":"[36]"},{"why":"Provides the Dutch supervised-finetuning data used in the posttraining stage shared across all ChocoLlama models.","marker":"[42]"},{"why":"MT-bench style side-by-side evaluation, the template ChocoLlama-Bench follows for judging multi-turn Dutch conversations.","marker":"[47]"},{"why":"Supplies the Dutch translations of ARC, HellaSwag, MMLU, and TruthfulQA used as the quantitative benchmark suite.","marker":"[23]"}],"fun_headline_variants":["LoRA teaches Llama-2 Dutch, but Llama-3 resists","Dutch tokenizer gives Llama-2 an edge, not Llama-3","Teaching Llama Dutch: tokenizer swap boosts Llama-2 only","Llama-2 learns Dutch via LoRA; Llama-3 already fluent","ChocoLlama: LoRA adaptation helps Llama-2, not Llama-3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparisons hinge on the paper's LLM-as-a-judge scores on ChocoLlama-Bench actually tracking Dutch generation quality, and on the GPT-3.5-translated multiple-choice questions remaining valid in Dutch; neither is checked against human ratings.","fun_headline_variants_meta":{"raw":{"variants":["LoRA teaches Llama-2 Dutch, but Llama-3 resists","Dutch tokenizer gives Llama-2 an edge, not Llama-3","Teaching Llama Dutch: tokenizer swap boosts Llama-2 only","Llama-2 learns Dutch via LoRA; Llama-3 already fluent","ChocoLlama: LoRA adaptation helps Llama-2, not Llama-3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":2030,"prompt_tokens":1128,"completion_tokens":902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":744,"completion_tokens_details":{"reasoning_tokens":795}},"tokens_in":744,"tokens_out":902,"duration_ms":6705,"temperature":1.0,"reasoning_tokens":795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:38:42.208609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 56-prompt side-by-side comparisons with human raters or a different judge model; if human preferences do not match the LLM-as-a-judge's preferences for the tokenizer-translated versus original-tokenizer ChocoLlama models, the claim that tokenizer modification improves language generation collapses. Alternatively, inspect the Dutch benchmark translations for translation errors that change the correct answer.","supporting_citations":[{"cited_title":"Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures","cited_arxiv_id":null,"evidence_quote":"Provides OSCAR, the source of 93GB of the Dutch pretraining data and the corpus used to train the Dutch BPE tokenizer."},{"cited_title":"Open multilingual llm evaluation leaderboard, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the Dutch translations of ARC, HellaSwag, MMLU, and TruthfulQA used as the quantitative benchmark suite."}],"review_version":1}