Fietje: An open, efficient LLM for Dutch

Bram Vanroy

Fietje: An open, efficient LLM for Dutch

T0 review · 0 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that Fietje, a 2.7-billion-parameter Dutch language model built by continue-pretraining Phi-2 on 28 billion Dutch tokens, is competitive with larger Dutch-specific models, with its chat version outperforming GEITje 7B…

desk verdict A transparent, reproducible Dutch SLM resource paper whose headline competitive claim rests on the two translated benchmarks; the artifacts are worth having, the ranking is shakier than the abstract suggests. read the letter →

arxiv 2412.15450 v1 pith:UCR3UWPX submitted 2024-12-19 cs.CL

Bram Vanroy This is my paper · ORCID

classification cs.CL

keywords Dutchlanguagemodelscontinuedpretrainingsmalllow-resourceNLPmodelevaluationreproducibilitymultilingualbenchmarkspreferenceoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a 2.7-billion-parameter model, built by continue-pretraining Phi-2 on 28 billion filtered Dutch tokens, can reach competitive quality for Dutch despite its small size. The stakes are practical: Dutch remains an underrepresented language in large language models, and Fietje is a fully open recipe, with weights, data, and training and evaluation code all public. The paper's central empirical finding is that the chat variant outperforms larger Dutch-specific models like GEITje 7B Ultra and Tweety on two of the five zero-shot benchmarks (ARC reasoning and Global MMLU). It also finds that recent small multilingual models outperform older and larger Dutch-adapted models, so release date and multilingual pretraining matter more than parameter count.

What carries the argument

The load-bearing object is the continued-pretraining pipeline: starting from the English-centric Phi-2, the model is trained on a curated 28-billion-token Dutch mixture of Wikipedia and heavily filtered web data, then supervised fine-tuned on Dutch instruction conversations, then aligned with Direct Preference Optimization on Dutch preference pairs. This is what carries the adaptation claim. The evaluation side is carried by a uniform zero-shot protocol with constrained label decoding and five repeated runs per model, which produces confidence intervals for the benchmark scores and is meant to make the comparisons reproducible.

What would settle it

Replace the two machine-translated benchmarks with human-written Dutch equivalents and add a native-speaker fluency judgment; if Fietje Chat then falls behind GEITje 7B Ultra and Tweety, the paper's competitive claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that continued pretraining can make a small, English-centric model genuinely Dutch: Fietje takes Phi-2, a 2.7-billion-parameter model, and trains it further on 28 billion Dutch tokens drawn from filtered Wikipedia and web data. From that base, an instruct version is trained on synthetic Dutch conversations and a chat version is aligned with Direct Preference Optimization. In the paper's evaluation, Fietje Chat beats GEITje 7B Ultra and Tweety, both 7-billion-parameter models built specifically for Dutch, on ARC and Global MMLU, and the author states that upon its release Fietje was the best Dutch model in its weight class. The same evaluation shows that newer multilingual small models such as Phi 3.5 and Qwen 2.5 have since gone past all Dutch-specific models tested, which the paper reads as evidence that multilingual pretraining and release date now dominate size.

Load-bearing premise

The load-bearing premise is that the five zero-shot benchmarks—two machine-translated and all of them label-selection tasks—reflect real Dutch language quality; the paper itself notes it does not measure fluency, so if those benchmarks are biased the competitive ranking could change.

Editorial extensions

If this is right

A 2.7-billion-parameter Dutch model can serve tasks that previously required 7B Dutch-specific models, lowering hardware and cost barriers for Dutch-language applications.
The public release of weights, data, and training and evaluation code makes Fietje a reusable template for adapting small LLMs to other underrepresented languages.
Future Dutch adaptation should start from current multilingual base models rather than older English-only ones, since newer small multilingual models already outperform older adapted 7B models.
Combining a Dutch-optimized tokenizer with continued pretraining is a likely next step, because tokenizer efficiency and corpus adaptation improve different aspects of Dutch processing.
Rankings built on translated benchmarks should be read with caution until Dutch-native benchmarks that also measure fluency are available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own limitation section implies that its label-choice benchmarks do not measure Dutch fluency; an open-ended generation test with native speakers would be a stronger test of the competitive claim.
The strict data filters used here probably remove some legitimate Dutch text, so a more nuanced quality classifier could unlock the larger 55-billion-token subset that the author also releases.
If continued pretraining is combined with vocabulary expansion, the resulting small model could plausibly close the remaining gap to large multilingual models on Dutch-specific tasks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bram,

Quick take: this is a useful resource paper, and the model itself is honestly framed as an intermediate step. The real value is the open stack: weights, the filtered 28B-token Dutch corpora, training configs, and a reproducible evaluation harness with confidence intervals. The writing is transparent about limitations, including the translationese problem and the absence of fluency measures.

What's new: the model family, the open corpus, and a 14-model comparison on five Dutch benchmarks. The method is not novel — continued pretraining, SFT, DPO — but the paper doesn't oversell it. It also makes a practical contribution by releasing the filtering heuristics and the benchmark code. That is worth credit.

The main soft spot is exactly what the stress-test flags. Fietje Chat's two wins over GEITje 7B Ultra and Tweety are on ARC and Global MMLU, the two machine-translated benchmarks. On all three native-Dutch benchmarks (DBRD, Dutch CoLA, XL WIC), Fietje Chat trails GEITje Ultra, and beats Tweety only on DBRD. So the 'competitive with larger Dutch-specific models' claim is fragile; the paper's own discussion phrasing ('comparable to or exceeding') is more careful, but the abstract and intro lean harder on the translated benchmarks. The reader's summary repeats that stronger formulation. That is a real caveat, but it is a caveat about how to read Table 2, not a load-bearing flaw in the resource contribution.

Second soft spot: base Fietje sometimes underperforms its own starting model Phi-2, which the paper acknowledges. That weakens any claim that continued pretraining helps in general, though the instruct/chat variants do improve substantially.

Third, minor: benchmarks are zero-shot only, one prompt per task, with temperature 1 sampling. The confidence intervals partly address that, but the ranking column invites over-reading.

Who is this for: anyone building Dutch NLP resources, especially practitioners who want a small, reproducible Dutch model and a benchmark harness to test their own. The paper will also interest people studying language adaptation for moderately low-resource languages.

My recommendation: send it to peer review. It deserves a serious referee despite the fragile comparative claim, because the artifacts and evaluation harness are reproducible and the limitations are stated in the text. The referee should push the authors to tighten the claim so it doesn't over-rely on the translated benchmarks. I would not cite the model as SOTA, but I would cite the corpus and benchmark suite if I worked on Dutch.

Referee Report

0 major / 7 minor

Summary. Fietje is a 2.8B-parameter Dutch language model obtained by continued pretraining Microsoft's Phi-2 on 28B Dutch tokens (Wikipedia plus filtered CulturaX), followed by an SFT-based instruct variant and a DPO-based chat variant. The paper describes data filtering, training configurations, and releases weights, data, and code. It evaluates the Fietje family and 11 comparison models on five zero-shot Dutch benchmarks (ARC, DBRD, Dutch CoLA, Global MMLU, XL WIC) with five repeated runs and reported confidence intervals. The headline results are that Fietje Chat is competitive for its size at the time of release, outperforming GEITje 7B Ultra and Tweety on two of five tasks (ARC and Global MMLU), while the paper also documents that newer multilingual models have surpassed it. The central contribution is the open and reproducible model/training pipeline rather than a new state of the art.

Significance. If the results hold, the main value is practical and infrastructural: the paper provides an open Dutch small language model with public training data, training configurations, and a reusable evaluation harness with confidence intervals, which is rare for Dutch. The paper is unusually transparent about limitations (zero-shot only, one prompt, no fluency metrics, possible translationese effects), and it explicitly warns against reading benchmark numbers as full evidence of Dutch fluency. The benchmark comparison is useful for practitioners, even though Fietje is no longer the best available Dutch/multilingual small model. The release of Dutch instruction and preference datasets from prior work is also a valuable community asset.

minor comments (7)

[Section 4.4, Fietje paragraph] The sentence that Fietje Chat surpasses GEITje Ultra and Tweety in two out of five tasks is correct, but those two wins are exactly the two translated/post-edited benchmarks (ARC and Global MMLU), while Fietje trails on DBRD, Dutch CoLA, and XL WIC. Adding this nuance next to the claim would prevent overreading, especially since Section 5 warns about translationese effects.
[Section 5] The sentence 'exhibiting results comparable to or exceeding GEITje 7B Ultra on benchmarks such as MMLU, Dutch CoLA, and ARC' is imprecise: Table 2 shows Fietje-2b-chat exceeds GEITje Ultra on Global MMLU and ARC but is lower on Dutch CoLA (45.45 vs 46.57). Please state the direction of each comparison or explicitly note the confidence-interval overlap.
[Sections 4.3 and 5] Global MMLU is not purely machine-translated; Section 4.3 says the Dutch portion was machine-translated and then improved by human post-editing. Please describe it as 'machine-translated with human post-editing' when calling it a translated benchmark in the Discussion.
[Appendix B.1] The continued-pretraining configuration points to a local scratch path (e.g., /dodrio/scratch/projects/2023_071/alignment-handbook/data/fietje-2b-cpt-prep) rather than a public dataset identifier. Since reproducibility is a stated contribution, please include the Hugging Face dataset path in the configuration or explain how to reconstruct the exact training split.
[Sections 2 and 3] There are several typos and heading issues that should be corrected: 'funetuning' should be 'fine-tuning', 'preference dating' should be 'preference data', the section headings 'T raining' should be 'Training', and 'Massive pretrain corpora' should be 'Massive pretraining corpora'.
[Table 1 caption and rows] The 'data transparency' column uses the value 'underspecified' for Phi-3.5-mini-instruct and Qwen2.5-3B-Instruct, but this term is not defined in the caption. Please define it, for example as 'partially described but not publicly released'.
[Section 4.1] The sentence about sampling says 'no top p or top k selection' with temperature 1, which is slightly ambiguous. Please state explicitly whether this means default top_p=1 and top_k=-1, or greedy decoding was not used at all.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: an empirical system paper with open data and independent benchmarks.

full rationale

Fietje is a system-description paper. The central claim (that Fietje, a continue-pretrained Phi-2, is competitive on Dutch benchmarks) is supported by scores in Table 2 on five external, pre-existing benchmarks: ARC, DBRD, Dutch CoLA, Global MMLU, and XL WIC-NL. These benchmarks were not constructed from Fietje's training data, and no parameter was fitted to them. The instruction and preference datasets (UltraChat 200K Dutch, No Robots Dutch, UltraFeedback Dutch Cleaned, Orca DPO Pairs Dutch Cleaned) were introduced in the author's prior work (Vanroy 2024) and are used only for training the Instruct and Chat variants, not for evaluation; this is a normal data provenance chain, not a circularity. The paper's caveat about machine-translated ARC and Global MMLU potentially favoring certain models (Sections 5 and 6.2) concerns external validity, not circular reasoning. No equation in the paper defines a predicted quantity in terms of the outcome being predicted, and no fitted input is relabeled as a prediction. The benchmark harness is open and reproducible, and the comparison models are independently evaluated under the same protocol. Consequently, the derivation chain is self-contained: training choices lead to weights, weights are scored on independent tests, and scores are reported as empirical results. The score of 1 reflects only the presence of minor self-citations for dataset provenance; none of them is load-bearing for the comparative evaluation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its dependency on hand-chosen hyperparameters and assumptions about benchmark validity and data quality is clearly disclosed but nonetheless load-bearing for the central claim.

free parameters (4)

DPO beta = 0.2
Chosen after hyperparameter tuning to reduce hallucinations and catastrophic forgetting (Section 3.3, Appendix B.3).
Base model learning rate = 9e-05
Hand-selected in the continued pretraining config (Appendix B.1).
SFT learning rate = 6e-05
Hand-selected in the instruction tuning config (Appendix B.2).
Data filter thresholds = punct>0.2, upper>0.22, digits>0.16, token len 2-20
Hand-picked heuristics based on SONAR-500 statistics (Section 3.1).

assumptions (4)

domain assumption Continued pretraining on a large Dutch corpus improves an English-centric SLM for Dutch tasks.
The entire approach assumes this transfer works; the paper's own results show the base Fietje sometimes underperforms Phi-2, so this is not fully borne out.
domain assumption The SONAR-500 corpus provides a valid baseline of high-quality Dutch text for setting filtering thresholds.
Section 3.1: thresholds were calculated on SONAR-500 and manually checked.
domain assumption The selected benchmarks (including machine-translated ARC and Global MMLU) measure relevant Dutch capabilities.
Section 4.3 and 6.2 discuss translationese effects and the lack of fluency metrics; the paper nonetheless uses these numbers as the primary evidence of competitiveness.
domain assumption Zero-shot, constrained decoding with sampling and five runs gives reliable performance estimates.
Section 4.1; the CIs are used to compare models, but only one prompt per task is used.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fietje: An open, efficient LLM for Dutch." pith.science (2026). https://pith.science/paper/UCR3UWPX

@misc{pith2026241215450,
  author       = {Pith},
  title        = {Pith review of: Fietje: An open, efficient LLM for Dutch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UCR3UWPX}},
  note         = {Machine review of arXiv:2412.15450}
}

read the original abstract

This paper introduces Fietje, a family of small language models (SLMs) specifically designed for the Dutch language. The model is based on Phi 2, an English-centric model of 2.7 billion parameters. Fietje demonstrated competitive results with larger language models upon its release. A core emphasis of this work is transparency and reproducibility: Fietje is fully open-source, with model weights, datasets, training, and evaluation code all publicly accessible. The paper discusses the performance of Fietje and many other models on an extensive evaluation suite of benchmarks on reasoning, sentiment analysis, world knowledge, linguistic acceptability and word sense disambiguation. Evaluation results illustrate the rapid progress in the field of LLMs, where recent small models outperform older, larger models that were fine-tuned for Dutch. This trend signals an exciting future for Dutch language processing, suggesting that even compact LLMs are becoming increasingly capable. Furthermore, ongoing and future efforts to adapt LLMs to Dutch are poised to enhance these models even further, broadening their applicability and accessibility. Fietje is only an intermediate step in improving accessibility to language technology for users of the Dutch language.

Figures

Figures reproduced from arXiv: 2412.15450 by the authors.

**Figure 1.** Model size vs. median performance However, when considering the release date of the models (Fig. 2a), it becomes clear that more recent models tend to have an advantage over older models (considering their respective model sizes). This becomes very clear when focusing on the un-adapted models in Figure 2b, where newer models like Phi 3.5 and Qwen 2.5 have an edge over Phi 2, and even over the much larger but older M… view at source ↗

**Figure 2.** Model release date vs. median performance [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

**Figure 3.** Results per benchmark ranking model, which is likely due to its varied Dutch-English dataset. A curious exception is its poor performance in the disambiguation task WIC, where it scores worst of all models (Fig. 3e). Mistral 7B Instruct’s surprising performance The Mistral 7B Instruct model demonstrates strong overall performance, with the notable exception of the Dutch CoLA benchmark (Fig. 3c), where it ranks last … view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance vs. size across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p032_4.png]

**Figure 5.** Figure 5: Performance vs. release date across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p033_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
cs.CL 2026-08 conditional novelty 6.0 of 10

A Dutch governmental LLM benchmark suite reveals consistent quality-cost-energy trade-offs and a dissociation between factuality and honesty across 31 models.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · cited by 1 Pith paper

[18]

steps" eval_steps: 900 save_strategy:

https://github.com/BramVanroy/fietje-2/tree/main/training # Training arguments learning_rate: 9.0e-05 adam_beta1: 0.9 adam_beta2: 0.98 adam_epsilon: 1.0e-7 weight_decay: 0.1 logging_steps: 1 logging_strategy: steps lr_scheduler_type: linear max_seq_length: 2048 per_device_train_batch_size: 40 per_device_eval_batch_size: 40 gradient_accumulation_steps: 3 g...

work page
[4237]

zak” and “uilskuiken

https://aclanthology.org/2024.lrec-main.377. Nielsen, Dan Saattrup (2023), ScandEval: A Benchmark for Scandinavian Natural Language Pro- cessing, Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 185–201. Penedo, Guilherme, Hynek Kydl ´ ıˇ cek, Vinko Sabolˇ cec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro vo...

arXiv 2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [18]

steps" eval_steps: 900 save_strategy:

https://github.com/BramVanroy/fietje-2/tree/main/training # Training arguments learning_rate: 9.0e-05 adam_beta1: 0.9 adam_beta2: 0.98 adam_epsilon: 1.0e-7 weight_decay: 0.1 logging_steps: 1 logging_strategy: steps lr_scheduler_type: linear max_seq_length: 2048 per_device_train_batch_size: 40 per_device_eval_batch_size: 40 gradient_accumulation_steps: 3 g...

work page

[2] [4237]

zak” and “uilskuiken

https://aclanthology.org/2024.lrec-main.377. Nielsen, Dan Saattrup (2023), ScandEval: A Benchmark for Scandinavian Natural Language Pro- cessing, Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 185–201. Penedo, Guilherme, Hynek Kydl ´ ıˇ cek, Vinko Sabolˇ cec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro vo...

arXiv 2023