Pith. sign in

REVIEW 4 major objections 6 minor 50 references

ChocoLlama: Lessons Learned From Teaching Llamas Dutch

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that continued pretraining with LoRA at 32B tokens can adapt Llama-2 to Dutch, that a Dutch-specific tokenizer with embedding reinitialization improves quality and efficiency, and that the same recipe adds little to the…

desk verdict Genuine empirical contribution with a solid Llama-2 recipe, but the Llama-3/posttraining conclusion is confounded and the self-built benchmark's reliability claim is unvalidated. read the letter →

arxiv 2412.07633 v1 pith:4KWDXAUG submitted 2024-12-10 cs.CL

Matthieu Meeus , Anthony Rathé , François Remy , Pieter Delobelle , Jens-Joris Decorte , Thomas Demeester This is my paper · ORCID
classification cs.CL
keywords languageadaptationDutchLoRAtokenizertranslationembeddingreinitializationcontinuedpretrainingmultilingualLLMevaluationChocoLlama-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports an attempt to adapt Llama-2 and Llama-3, two English-centric large language models, to Dutch. The central claim is that continued pretraining with Low-Rank Adaptation (LoRA), updating only about 8-12% of parameters during pretraining on 32 billion Dutch tokens, can successfully teach Llama-2 Dutch. A second claim is that replacing Llama-2's tokenizer with a Dutch-trained one, plus reinitializing embeddings to match old tokens, improves quality and reduces token count by 29.4%. The paper also argues that the same continued-pretraining recipe gives only limited gains for Llama-3, which already has strong Dutch abilities, concluding that for increasingly multilingual foundation models, language-specific posttraining may matter more than continued pretraining. These findings matter because they suggest a cost-effective recipe for adapting existing models to lower-resource languages, and they refine expectations about when further pretraining is worth the effort.

What carries the argument

The two load-bearing mechanisms are (1) LoRA applied at scale, training low-rank updates on all target modules (r=8, alpha=32) with full updates to the embedding and output head, giving 544M-1.07B trainable parameters (7.75-11.77% of each model); and (2) tokenizer translation with embedding reinitialization, in which a new Dutch BPE tokenizer (50k tokens) replaces the original and each new token's embedding is initialized from a semantically similar old token before continued pretraining. The paper also introduces a normalized-perplexity formula (Equation 1) that rescales token-level perplexity to a common reference tokenizer, allowing fair comparison of training curves across different tokenization schemes.

What would settle it

Run the same 56-prompt side-by-side comparisons with human raters or a different judge model; if human preferences do not match the LLM-as-a-judge's preferences for the tokenizer-translated versus original-tokenizer ChocoLlama models, the claim that tokenizer modification improves language generation collapses. Alternatively, inspect the Dutch benchmark translations for translation errors that change the correct answer.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is twofold. First, a parameter-efficient continued-pretraining scheme, in which LoRA adapters are trained on all attention and MLP layers while the embedding and head are fully updated, is sufficient to make Llama-2 respond in fluent Dutch: the adapted models beat the original Llama-2 on both standard multiple-choice benchmarks and the new ChocoLlama-Bench conversation benchmark. Second, translating the tokenizer, replacing the English-trained vocabulary with a Dutch-trained BPE vocabulary of 50,000 tokens and initializing each new embedding from a semantically similar old token, yields a further gain: the tokentrans model beats the original-tokenizer model, particularly on writing tasks, and compresses Dutch text into 29.4% fewer tokens. When the same recipe is applied to Llama-3, which was released mid-project and already outperformed the adapted Llama-2, the continued pretraining produces a model that wins on multiple-choice benchmarks but loses on ChocoLlama-Bench against the original Llama-3-instruct; the paper reads this as evidence that continued pretraining yields diminishing returns for models with strong multilingual capabilities.

Load-bearing premise

The comparisons hinge on the paper's LLM-as-a-judge scores on ChocoLlama-Bench actually tracking Dutch generation quality, and on the GPT-3.5-translated multiple-choice questions remaining valid in Dutch; neither is checked against human ratings.

Editorial extensions

If this is right

  • Continued pretraining with LoRA at the 32B-token scale is a viable, low-cost route for adapting primarily English models to a new language: about 10% of parameters are trained, and the resulting models use Dutch consistently.
  • A language-specific tokenizer with embedding reinitialization not only improves Dutch generation quality for Llama-2 but cuts token count by 29.4%, reducing training and inference cost.
  • For models like Llama-3 that already have broad multilingual coverage, further Dutch pretraining gives at best marginal gains on standard benchmarks and a small regression on conversational quality, so the bottleneck shifts to posttraining.
  • The open ChocoLlama-Bench (56 Dutch prompts across reasoning, comprehension, and writing, scored by an LLM judge) provides a new resource for comparing Dutch generation quality beyond multiple-choice accuracy.
  • The qualitative finding that the original Llama-2 often answers in English while the adapted models answer in Dutch suggests the adaptation genuinely changes the model's language behavior, not just benchmark scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LLM-as-a-judge scores on ChocoLlama-Bench are even approximately right, the tokenizer-translation result has a strong practical corollary: adapting the tokenizer should be considered for any language whose tokenization differs substantially from English, and the same embedding-reinitialization trick could be combined with other parameter-efficient methods.
  • The Llama-3 result suggests a testable hypothesis the paper leaves implicit: continued pretraining on a language already well represented in the base model may be near-neutral or even harmful for conversational quality, and posttraining on high-quality native data is the higher-leverage intervention.
  • The 29.4% token reduction implies an inference-cost benefit that the paper does not quantify; one could measure tokens per Dutch sentence for the two ChocoLlama-2 variants to estimate speed and memory savings in practice.
  • Because the posttraining data used in the paper cannot be used commercially, the net practical value of the recipe depends on retraining the SFT and DPO stages with permissive data; a reader could test whether the base-model gains survive a fully open posttraining pipeline.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper adapts Llama-2 and Llama-3 to Dutch through LoRA-based continued pretraining on 104 GB of Dutch text (32B Llama-2 tokens), optionally replacing the tokenizer with a Dutch-specific BPE tokenizer plus embedding reinitialization, and then applying SFT and DPO from prior work. The authors evaluate the resulting ChocoLlama models on Dutch translations of ARC, HellaSwag, MMLU, and TruthfulQA, and on a new benchmark (ChocoLlama-Bench) in which GPT-4o judges side-by-side responses to 56 Dutch prompts. They report that LoRA-based continued pretraining improves Llama-2's Dutch performance, that the Dutch-tokenizer variant outperforms the original-tokenizer variant, and that the same recipe yields limited gains for Llama-3, leading to the conclusion that language adaptation for multilingual foundation models should focus on posttraining rather than continued pretraining.

Significance. The work is practically valuable: it releases models, code, and a new Dutch benchmark, and it applies trans-tokenization with embedding reinitialization to autoregressive models at a scale (32B tokens) not previously demonstrated. If the claims hold, the paper would be a useful case study for low-resource language adaptation with parameter-efficient methods. The main significance, however, hinges on the reliability of the evaluation. The paper's key qualitative conclusions—especially the claim that continued pretraining yields limited gains for Llama-3—are drawn from an unvalidated LLM-as-a-judge benchmark, and the quantitative deltas in the standard benchmarks are small and lack statistical support. These issues prevent the headline lessons from being established as stated, though they are addressable with additional experiments and analysis.

major comments (4)
  1. [Section 6 and Section 5.2.1] The paper asserts that ChocoLlama-Bench 'offer[s] a more comprehensive and reliable evaluation' (Section 6) and that this reliability is 'empirically' found, but no evidence is provided: there is no human evaluation, no inter-annotator agreement, no comparison with the quantitative benchmarks, and no analysis of GPT-4o judge bias. This matters because the benchmark is used to overturn the quantitative result in Table 2, where Llama-3-ChocoLlama-8B-instruct outperforms Meta-Llama-3-8B-Instruct on average (0.53 vs 0.51). The central conclusion that continued pretraining adds little for Llama-3 depends critically on this unvalidated judge, so the reliability claim is load-bearing and must be supported.
  2. [Section 5.2.4] The comparison underlying the headline lesson 'posttraining rather than continued pretraining' is confounded. Llama-3-ChocoLlama-8B-instruct differs from Meta-Llama-3-8B-Instruct in two variables: whether Dutch continued pretraining was performed and which posttraining recipe was used (Vanroy/Zephyr SFT+DPO vs Meta's own instruction tuning). The paper itself acknowledges (Section 5.2.4) that Meta's instruction tuning is likely better than the adopted recipe, so the observed ChocoLlama-Bench outcome cannot isolate the effect of continued pretraining. A same-posttraining ablation, such as applying the same Dutch posttraining to both Meta-Llama-3-8B and Meta-Llama-3-8B-Instruct, or applying Meta's posttraining to Llama-3-ChocoLlama-base, is needed before drawing the stated conclusion.
  3. [Section 2.2, Table 1, and Table 2] The comparison between the original-tokenizer model (ChocoLlama-2-7B-base) and the Dutch-tokenizer model (ChocoLlama-2-7B-tokentrans-base) is confounded by training duration and token count: Table 1 shows the tokentrans model was trained for 43.8k steps on 22.6B tokens, whereas the original-tokenizer model was trained for 62.1k steps on 32B tokens. Because both are trained for exactly one epoch, the new tokenizer changes both the representation and the number of gradient updates. To attribute the observed improvement (e.g., 0.45 vs 0.41 average in Table 2) to tokenizer modification with weight reinitialization, one would need to control for steps or tokens, for example by training the original-tokenizer model for the same number of steps or the trans-tokenized model for the same number of tokens.
  4. [Section 5.1 and Table 3] The quantitative gains are small and no measures of uncertainty or significance are reported: the best ChocoLlama-2-instruct average is 0.46 versus 0.43 for the original-tokenizer variant, and the improvement over Llama-2-7b-chat is 0.05; the per-benchmark differences are often within a few points (e.g., HellaSwag 0.62 vs 0.57, MMLU 0.34 vs 0.33). Similarly, the head-to-head result in Table 3 (24 wins vs 15 wins, with 17 ties, out of 56 turns) is not statistically significant under a simple binomial test on non-tied turns. Without confidence intervals, standard errors, or a significance test, the claims that tokenizer modification 'improves' performance and that LoRA 'effectively scales' rest on effects that could be noise. At minimum, the authors should report variance across benchmark items or repeated evaluations, or temper the strength of the claims.
minor comments (6)
  1. [Equation (1)] The notation in Equation (1) is inconsistent: the formula uses Pθ(Dbatch) and |Tθ(Dref)|/|Tθref(Dref)|, but the surrounding text says Pθ is the perplexity evaluated on Dref. Please clarify which dataset is used in the numerator and why the batch index appears in one place and Dref in another.
  2. [Section 5.1] The quantitative benchmarks use GPT-3.5-turbo Dutch translations of ARC, HellaSwag, MMLU, and TruthfulQA, but the paper does not discuss or validate the quality of these translations. A brief comment on translation fidelity, or a reference to prior validation, would help the reader judge whether the multiple-choice scores are meaningful for Dutch.
  3. [Table 3] The two comparisons against Llama-2-7b-chat give seemingly inconsistent strengths: ChocoLlama-2-7B-instruct wins 78% of turns, while ChocoLlama-2-7B-tokentrans-instruct wins only 66%, yet the tokentrans model beats the nontokentrans model in the direct comparison. This is possible because of ties, but the paper does not discuss the pattern; reporting only win percentages can be misleading when tie rates differ.
  4. [References] References [46] and [47] are the same paper (Zheng et al., MT-Bench / Chatbot Arena); one should be removed or both should be merged.
  5. [Section 5.2.1] The description of ChocoLlama-Bench prompts as '30 Dutch prompts and follow-up prompts' is ambiguous; it would be clearer to say 30 multi-turn conversations comprising 56 turns total.
  6. [Section 6] The phrase 'we empirically find this benchmark to offer a more comprehensive and reliable evaluation' is not supported by any empirical comparison; consider replacing 'empirically' with a direct description of what was measured, or softening the claim until supporting evidence is added.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the Llama-2 results are direct empirical comparisons, and the weaker Llama-3 'posttraining' lesson is a confounded inference plus an unvalidated benchmark-reliability claim, not a definitional or self-citational reduction.

full rationale

This paper is an empirical adaptation study rather than a derivation, so the equation-level circularity tests mostly do not apply. The two headline Llama-2 results (LoRA-based continued pretraining improves Dutch capabilities; a Dutch tokenizer plus embedding reinitialization improves on the original tokenizer) are supported by direct side-by-side comparisons against the unadapted Llama-2 on external translated benchmarks (Table 2) and on ChocoLlama-Bench (Table 3). The tokenizer-reinitialization method is attributed to Remy et al. [32], which has overlapping authorship, but the central claim is not imported from that citation: the paper's own comparison between ChocoLlama-2-7B-instruct and ChocoLlama-2-7B-tokentrans-instruct is the evidence, with [32] cited for provenance rather than as the proof. The Llama-3 lesson ('posttraining rather than continued pretraining') is weaker: it compares Llama-3-ChocoLlama-8B-instruct with Meta-Llama-3-8B-Instruct across two varying factors (continued pretraining and posttraining recipe), and Section 5.2.4 explicitly concedes that 'the instruction tuning applied to Meta-Llama-3-8B-Instruct is likely to be significantly better than the one we here adopt from prior work.' That is a confounded inference. Likewise, Section 6 asserts that ChocoLlama-Bench offers 'a more comprehensive and reliable evaluation' without external validation, while also acknowledging that GPT-4o 'may introduce its own biases and limitations'; these are evaluation-validity and experimental-design concerns, not a reduction of the conclusion to the benchmark construction. No fitted parameter is renamed as a prediction, no result is defined in terms of its own target, and no load-bearing argument collapses into a self-citation chain. Score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central empirical claims rest on several assumptions that are not proven in the paper: normalized perplexity comparability, validity of the LLM judge, preservation of benchmark validity under translation, and transfer of embedding-reinitialization to autoregressive models. The hyperparameters are chosen by hand; no tuning or ablations are reported.

free parameters (3)
  • LoRA hyperparameters = r=8, alpha=32
    Chosen by hand; no tuning or ablation is reported. They control the rank and scale of the low-rank updates, so the capacity for language adaptation depends on them.
  • Dutch tokenizer vocabulary size = 50,000 tokens
    Selected by the authors; larger than Llama-2's 32k vocabulary. It directly affects compression (29.4% fewer tokens) and trainable parameter count, but no ablation over vocabulary size is given.
  • Pretraining schedule and batch size = lr=3e-4, gamma=0.85 every 10%, batch=128 or 512, context=4096
    Chosen by hand and not tuned; they can affect final model quality and perplexity curves.
assumptions (4)
  • domain assumption Equation (1) normalized perplexity is comparable across different tokenizers (Mielke 2019)
    Used in Section 2.3 to compare ChocoLlama-2-base and tokentrans-base and to conclude the tokenizer variant catches up. If the normalization is not valid, the perplexity comparison is meaningless.
  • domain assumption GPT-4o provides valid preference judgments for Dutch text
    The new benchmark's scores and the main comparisons in Section 5.2 depend on this. The paper acknowledges bias risk but does not validate the judge against human ratings.
  • domain assumption Dutch translations of ARC, HellaSwag, MMLU, TruthfulQA preserve what the original benchmarks measure
    Quantitative results in Table 2 rest on these translations from the Open Dutch LLM Leaderboard; translation artifacts could affect accuracy.
  • domain assumption Embedding reinitialization method from Remy et al. [32] transfers from masked LMs to autoregressive Llama-2
    The tokenizer variant relies on this method; the paper tests it and finds it works, but the premise is from prior work.
invented entities (1)
  • ChocoLlama-Bench
    purpose: New 56-turn Dutch conversation benchmark used to rank models and to support the conclusion that Llama-3-instruct beats the adapted Llama-3.
    The paper provides no human evaluation or correlation with other benchmarks, so the benchmark has no independent validation; the reliability claim is self-referential.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChocoLlama: Lessons Learned From Teaching Llamas Dutch." pith.science (2026). https://pith.science/paper/4KWDXAUG

@misc{pith2026241207633,
  author       = {Pith},
  title        = {Pith review of: ChocoLlama: Lessons Learned From Teaching Llamas Dutch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KWDXAUG}},
  note         = {Machine review of arXiv:2412.07633}
}
abstract

While Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation, their performance often lags in lower-resource, non-English languages due to biases in the training data. In this work, we explore strategies for adapting the primarily English LLMs (Llama-2 and Llama-3) to Dutch, a language spoken by 30 million people worldwide yet often underrepresented in LLM development. We collect 104GB of Dutch text ($32$B tokens) from various sources to first apply continued pretraining using low-rank adaptation (LoRA), complemented with Dutch posttraining strategies provided by prior work. For Llama-2, we consider using (i) the tokenizer of the original model, and (ii) training a new, Dutch-specific tokenizer combined with embedding reinitialization. We evaluate our adapted models, ChocoLlama-2, both on standard benchmarks and a novel Dutch benchmark, ChocoLlama-Bench. Our results demonstrate that LoRA can effectively scale for language adaptation, and that tokenizer modification with careful weight reinitialization can improve performance. Notably, Llama-3 was released during the course of this project and, upon evaluation, demonstrated superior Dutch capabilities compared to our Dutch-adapted versions of Llama-2. We hence apply the same adaptation technique to Llama-3, using its original tokenizer. While our adaptation methods enhanced Llama-2's Dutch capabilities, we found limited gains when applying the same techniques to Llama-3. This suggests that for ever improving, multilingual foundation models, language adaptation techniques may benefit more from focusing on language-specific posttraining rather than on continued pretraining. We hope this work contributes to the broader understanding of adapting LLMs to lower-resource languages, and to the development of Dutch LLMs in particular.

Figures

Figures reproduced from arXiv: 2412.07633 by the authors.

Figure 1
Figure 1. Hand-picked conversational snippet of Llama-3-ChocoLlama-instruct. Answers to the same prompt for other models considered in this paper can be found in Appendix A. Crawl [7] represents the majority of the training data for GPT-3 [5], Llama [38] and BLOOM [35]. Textual data on the web, however, represents a highly skewed language distribution. OSCAR [36] classifies Common Crawl [7] by language and finds that more tha… view at source ↗
Figure 2
Figure 2. Model perplexity Pθ(Dbatch) (left) and normalized perplexity P norm θ (Dbatch) (see Equation 1) (right) across the ChocoLlama model suite during pretraining over 1 full epoch of the collected Dutch data. While the observed trends in training perplexity provide insights into the models’ learning progress, it is important to note that the perplexity values are not directly comparable across models due to differences i… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 25 canonical work pages

  1. [1]

    Llama recipes

    Meta AI. Llama recipes. https://github.com/meta-llama/llama-recipes, 2023

    work page 2023

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md

    2024

  3. [3]

    A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference o...

    work page 2023

  4. [4]

    Bert-nl a set of language models pre-trained on the dutch sonar corpus

    A Brandsen, A Dirkson, S Verberne, M Sappelli, D Manh Chu, and K Stoutjesdijk. Bert-nl a set of language models pre-trained on the dutch sonar corpus. 2019

    work page 2019

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  6. [6]

    Think you have solved question answering? try arc, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457

    arXiv 2018

  7. [7]

    Common crawl

    Common Crawl. Common crawl. https://commoncrawl.org

    work page

  8. [8]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm

    2023

Show all 50 references
  1. [9]

    Bertje: A dutch bert model

    Wietse De Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. Bertje: A dutch bert model. arXiv preprint arXiv:1912.09582 , 2019

    1912 arXiv

  2. [10]

    Robbert: a dutch roberta-based language model

    Pieter Delobelle, Thomas Winters, and Bettina Berendt. Robbert: a dutch roberta-based language model. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3255–3265, 2020

    2020

  3. [11]

    Reynaerde-7b-chat model card

    Julien Van den Avenne. Reynaerde-7b-chat model card. 2024. URL https://huggingface.co/ ReBatch/Reynaerde-7B-Chat

    2024

  4. [12]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36, 2024

    2024

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018

    2018 arXiv

  6. [14]

    How to adapt your pretrained multilingual model to 1600 languages

    Abteen Ebrahimi and Katharina Kann. How to adapt your pretrained multilingual model to 1600 languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume ...

    2021

  7. [15]

    Croissantllm: A truly bilingual french-english language model

    Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, Ant´ onio Loison, Duarte Alves, Caio Corro, Nicolas Boizard, Jo˜ ao Alves, Ricardo Rei, Pedro Martins, et al. Croissantllm: A truly bilingual french-english language model. arXiv preprint arXiv:2402.00786 , 2024. 14

    2024 arXiv

  8. [16]

    ”wikimedia downloads”

    ”Wikimedia Foundation”. ”wikimedia downloads”. https://dumps.wikimedia.org

  9. [17]

    Project gutenberg, 2004

    Michael Hart. Project gutenberg, 2004. URL https://www.gutenberg.org/

    2004

  10. [18]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR, abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300

    2009 arXiv

  11. [19]

    Orpo: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691 , 2(4):5, 2024

    2024 arXiv

  12. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021

    2021

  13. [21]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023

    arXiv

  14. [22]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024

    2024 arXiv

  15. [23]

    Open multilingual llm evaluation leaderboard, 2023

    Viet Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Franck Dernoncourt, and Thien Huu Nguyen. Open multilingual llm evaluation leaderboard, 2023

    2023

  16. [24]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. CoRR, abs/2109.07958, 2021. URL https://arxiv.org/abs/2109.07958

    2021 arXiv

  17. [25]

    Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles

    Pierre Lison and J¨ org Tiedemann. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. 2016

    2016

  18. [26]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  19. [27]

    Can you compare perplexity across different segmentations

    Sabrina J Mielke. Can you compare perplexity across different segmentations. Available in: http://sjmielke. com/comparing-perplexities. htm, 2019

    2019

  20. [28]

    Gpt-4 technical report

    OpenAI2023. Gpt-4 technical report. https://cdn.openai.com/papers/gpt-4.pdf, 2023

    2023

  21. [29]

    Gutenberg scraper

    Katherine Pully. Gutenberg scraper. https://github.com/kpully/gutenberg_scraper, 2020

    2020

  22. [30]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024

    2024

  23. [31]

    Llama-3-8b-dutch model card

    ReBatch. Llama-3-8b-dutch model card. 2024. URL https://huggingface.co/ReBatch/ Llama-3-8B-dutch

    2024

  24. [32]

    Tik-to-tok: Translating language models one token at a time: An embedding initialization strategy for efficient language adaptation

    Fran¸ cois Remy, Pieter Delobelle, Bettina Berendt, Kris Demuynck, and Thomas Demeester. Tik-to-tok: Translating language models one token at a time: An embedding initialization strategy for efficient language adaptation. arXiv preprint arXiv:2310.03477 , 2023

    arXiv

  25. [33]

    Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of LLMs for low-resource NLP

    Fran¸ cois Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of LLMs for low-resource NLP. In First Conference on Language Modeling , 2024. UR...

    2024

  26. [34]

    Geitje: een groot open nederlands taalmodel, December

    Edwin Rijgersberg and Bob Lucassen. Geitje: een groot open nederlands taalmodel, December

  27. [35]

    Bloom: A 176b- parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagn´ e, Alexandra Sasha Luccioni, Fran¸ cois Yvon, Matthias Gall´ e, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022

    2022 arXiv

  28. [36]

    Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures

    Pedro Javier Ortiz Su´ arez, Beno ˆ ıt Sagot, and Laurent Romary. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) . Leibniz-Institut f¨ ur Deutsche Sprache, 2019

    2019

  29. [37]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  30. [38]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  32. [40]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://github.com/ huggingface/alignment-handbook, 2023

    2023

  33. [41]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl´ ementine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023

    2023

  34. [42]

    Language resources for dutch large language modelling

    Bram Vanroy. Language resources for dutch large language modelling. arXiv preprint arXiv:2312.12852, 2023

    2023 arXiv

  35. [43]

    Geitje-7b-ultra model card

    Bram Vanroy. Geitje-7b-ultra model card. 2024. URL https://huggingface.co/BramVanroy/ GEITje-7B-ultra

    2024

  36. [44]

    Adapting bigscience multilingual model to unseen languages

    Zheng-Xin Yong and Vassilina Nikoulina. Adapting bigscience multilingual model to unseen languages. arXiv preprint arXiv:2204.04873 , 2022

    2022 arXiv

  37. [45]

    Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830, 2019. URL http://arxiv.org/ abs/1905.07830

    1905 arXiv

  38. [46]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  39. [47]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/ 2306.056...

    2023 arXiv

  40. [49]

    System prompt: ”Je ben een expert in Vlaamse cultuur en je antwoordt behulpzaam, in het Nederlands, op de vraag van de gebruiker.”

  41. [50]

    Waarover zouden ze babbelen?” We further generate a maximum of 256 tokens, and use a temperature of 0 .6 and top-p of 0 .9 for generation

    User prompt: ”Jacques brel, Willem Elsschot en Jan Jambon zitten op caf´ e. Waarover zouden ze babbelen?” We further generate a maximum of 256 tokens, and use a temperature of 0 .6 and top-p of 0 .9 for generation. Tables 6 and 7 summarize the responses across models. While th...

  42. [2023]

    URL https://github.com/Rijgersberg/GEITje

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.