{"id":"d6c40ac1-9194-4dab-a23e-c6527b8fda5a","arxiv_id":"2412.15450","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fietje, a fully open 2.8-billion-parameter Dutch SLM family derived from Phi-2, achieves competitive zero-shot benchmark scores in its size class, though newer multilingual models already surpass it.","lead":"Fietje is a family of small Dutch language models built by continue-pretraining Microsoft's 2.8 billion parameter Phi-2 on 28 billion Dutch tokens, then adding instruction-tuned and chat versions. The paper reports benchmark results showing the chat version competes with larger Dutch-specific models, and releases weights, data, and code for full reproduction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fietje Chat's two benchmark wins over GEITje 7B Ultra and Tweety are exactly the two machine-translated benchmarks (ARC and Global MMLU); on all three native-Dutch benchmarks it trails GEITje and beats Tweety only on DBRD.","rationale":"The paper is a thorough, transparent system description: it releases model weights, training data, configuration files, and evaluation code, and its benchmark methodology is carefully documented with confidence intervals. The independent support from reproducibility is real and should count heavily. However, the strongest formulation of the central claim — that Fietje Chat outperforms larger Dutch-specific models on two of five benchmarks — relies on the exact two benchmarks that are machine-translated (ARC via GPT-3.5-turbo, Global MMLU via MT with human post-editing). The reader's weakest_assumption already identified benchmark validity as the soft spot, and my read agrees but sharpens it: the pattern in Table 2 is not just a hypothetical risk, it is visible in the data. The native-Dutch benchmarks tell a markedly different story, so the competitive claim as stated overreaches. Because the paper honestly discloses this limitation in Section 6.2 and the model release itself is a valuable contribution, rejection is not warranted. A conditional acceptance — requiring the authors to either add a native-only comparison or explicitly re-scope the claim to translated benchmarks — would align the verdict with the evidence.","tokens_in":50963,"tokens_out":7412,"duration_ms":61778,"concrete_test":"Recompute from Table 2 the head-to-head win counts between Fietje-2b-chat and each Dutch-specific comparator (GEITje-7B-ultra, tweety-7b-dutch-v24a), separated into the three original-Dutch benchmarks (DBRD, Dutch CoLA, XL WIC) and the two machine-translated benchmarks (ARC, Global MMLU). If the counts are 0/3 and 1/3 on native benchmarks versus 2/2 and 1/2 on translated benchmarks, then the paper's 'outperforms on two of five' claim is an artifact of benchmark selection and should be re-scoped or qualified in the final version.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparative claim in the reader's strongest_claim is that Fietje Chat outperforms larger Dutch-specific models on ARC and Global MMLU (two of five benchmarks). Table 2 shows this advantage is entirely confined to translated tasks: Fietje-2b-chat beats GEITje-7B-ultra on ARC (31.56 vs 29.10) and Global MMLU (26.36 vs 24.39), and beats tweety-7b-dutch-v24a on ARC (31.56 vs 29.46) but actually trails on Global MMLU (26.36 vs 27.36). On the three benchmarks based on original Dutch text (DBRD, Dutch CoLA, XL WIC), Fietje-2b-chat loses to GEITje-ultra on all three (58.78 vs 90.00, 45.45 vs 46.57, 39.24 vs 44.45) and loses to Tweety on Dutch CoLA and XL WIC (45.45 vs 51.27, 39.24 vs 43.23), only winning DBRD (58.78 vs 40.22). Thus the paper's headline, as summarized by the reader, depends on the two benchmarks most subject to translationese effects, which Sections 5 and 6.2 themselves acknowledge can bias comparisons. The paper's own 'comparable to or exceeding' phrasing in the Discussion is more prudent, but the strongest formulation of the claim is fragile: if translated benchmarks reward models with different training-data characteristics, the demonstrated outperformance does not establish broad Dutch-language competitiveness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Fietje is a 2.8B-parameter Dutch language model obtained by continued pretraining Microsoft's Phi-2 on 28B Dutch tokens (Wikipedia plus filtered CulturaX), followed by an SFT-based instruct variant and a DPO-based chat variant. The paper describes data filtering, training configurations, and releases weights, data, and code. It evaluates the Fietje family and 11 comparison models on five zero-shot Dutch benchmarks (ARC, DBRD, Dutch CoLA, Global MMLU, XL WIC) with five repeated runs and reported confidence intervals. The headline results are that Fietje Chat is competitive for its size at the time of release, outperforming GEITje 7B Ultra and Tweety on two of five tasks (ARC and Global MMLU), while the paper also documents that newer multilingual models have surpassed it. The central contribution is the open and reproducible model/training pipeline rather than a new state of the art.","tokens_in":51294,"tokens_out":8132,"duration_ms":74064,"significance":"If the results hold, the main value is practical and infrastructural: the paper provides an open Dutch small language model with public training data, training configurations, and a reusable evaluation harness with confidence intervals, which is rare for Dutch. The paper is unusually transparent about limitations (zero-shot only, one prompt, no fluency metrics, possible translationese effects), and it explicitly warns against reading benchmark numbers as full evidence of Dutch fluency. The benchmark comparison is useful for practitioners, even though Fietje is no longer the best available Dutch/multilingual small model. The release of Dutch instruction and preference datasets from prior work is also a valuable community asset.","major_comments":[],"minor_comments":[{"comment":"The sentence that Fietje Chat surpasses GEITje Ultra and Tweety in two out of five tasks is correct, but those two wins are exactly the two translated/post-edited benchmarks (ARC and Global MMLU), while Fietje trails on DBRD, Dutch CoLA, and XL WIC. Adding this nuance next to the claim would prevent overreading, especially since Section 5 warns about translationese effects.","section":"Section 4.4, Fietje paragraph"},{"comment":"The sentence 'exhibiting results comparable to or exceeding GEITje 7B Ultra on benchmarks such as MMLU, Dutch CoLA, and ARC' is imprecise: Table 2 shows Fietje-2b-chat exceeds GEITje Ultra on Global MMLU and ARC but is lower on Dutch CoLA (45.45 vs 46.57). Please state the direction of each comparison or explicitly note the confidence-interval overlap.","section":"Section 5"},{"comment":"Global MMLU is not purely machine-translated; Section 4.3 says the Dutch portion was machine-translated and then improved by human post-editing. Please describe it as 'machine-translated with human post-editing' when calling it a translated benchmark in the Discussion.","section":"Sections 4.3 and 5"},{"comment":"The continued-pretraining configuration points to a local scratch path (e.g., /dodrio/scratch/projects/2023_071/alignment-handbook/data/fietje-2b-cpt-prep) rather than a public dataset identifier. Since reproducibility is a stated contribution, please include the Hugging Face dataset path in the configuration or explain how to reconstruct the exact training split.","section":"Appendix B.1"},{"comment":"There are several typos and heading issues that should be corrected: 'funetuning' should be 'fine-tuning', 'preference dating' should be 'preference data', the section headings 'T raining' should be 'Training', and 'Massive pretrain corpora' should be 'Massive pretraining corpora'.","section":"Sections 2 and 3"},{"comment":"The 'data transparency' column uses the value 'underspecified' for Phi-3.5-mini-instruct and Qwen2.5-3B-Instruct, but this term is not defined in the caption. Please define it, for example as 'partially described but not publicly released'.","section":"Table 1 caption and rows"},{"comment":"The sentence about sampling says 'no top p or top k selection' with temperature 1, which is slightly ambiguous. Please state explicitly whether this means default top_p=1 and top_k=-1, or greedy decoding was not used at all.","section":"Section 4.1"}],"recommendation":"minor_revision","confidential_remarks":"The paper is a system/resource description by the model's author. The self-comparison with GEITje, on which the author has also worked, is handled transparently, and the limitations are candid. The main risk is that readers overinterpret the two wins on translated benchmarks; the authors can fix this with a clarifying sentence in Section 4.4. I see no scope or novelty problem for a journal that publishes language-resource papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bram,\n\nQuick take: this is a useful resource paper, and the model itself is honestly framed as an intermediate step. The real value is the open stack: weights, the filtered 28B-token Dutch corpora, training configs, and a reproducible evaluation harness with confidence intervals. The writing is transparent about limitations, including the translationese problem and the absence of fluency measures.\n\nWhat's new: the model family, the open corpus, and a 14-model comparison on five Dutch benchmarks. The method is not novel — continued pretraining, SFT, DPO — but the paper doesn't oversell it. It also makes a practical contribution by releasing the filtering heuristics and the benchmark code. That is worth credit.\n\nThe main soft spot is exactly what the stress-test flags. Fietje Chat's two wins over GEITje 7B Ultra and Tweety are on ARC and Global MMLU, the two machine-translated benchmarks. On all three native-Dutch benchmarks (DBRD, Dutch CoLA, XL WIC), Fietje Chat trails GEITje Ultra, and beats Tweety only on DBRD. So the 'competitive with larger Dutch-specific models' claim is fragile; the paper's own discussion phrasing ('comparable to or exceeding') is more careful, but the abstract and intro lean harder on the translated benchmarks. The reader's summary repeats that stronger formulation. That is a real caveat, but it is a caveat about how to read Table 2, not a load-bearing flaw in the resource contribution.\n\nSecond soft spot: base Fietje sometimes underperforms its own starting model Phi-2, which the paper acknowledges. That weakens any claim that continued pretraining helps in general, though the instruct/chat variants do improve substantially.\n\nThird, minor: benchmarks are zero-shot only, one prompt per task, with temperature 1 sampling. The confidence intervals partly address that, but the ranking column invites over-reading.\n\nWho is this for: anyone building Dutch NLP resources, especially practitioners who want a small, reproducible Dutch model and a benchmark harness to test their own. The paper will also interest people studying language adaptation for moderately low-resource languages.\n\nMy recommendation: send it to peer review. It deserves a serious referee despite the fragile comparative claim, because the artifacts and evaluation harness are reproducible and the limitations are stated in the text. The referee should push the authors to tighten the claim so it doesn't over-rely on the translated benchmarks. I would not cite the model as SOTA, but I would cite the corpus and benchmark suite if I worked on Dutch.","headline":"A transparent, reproducible Dutch SLM resource paper whose headline competitive claim rests on the two translated benchmarks; the artifacts are worth having, the ranking is shakier than the abstract suggests.","tokens_in":51828,"tokens_out":1935,"would_cite":true,"duration_ms":18298,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that Fietje, a 2.7-billion-parameter Dutch language model built by continue-pretraining Phi-2 on 28 billion Dutch tokens, is competitive with larger Dutch-specific models, with its chat version outperforming GEITje 7B…","keywords":["Dutch language models","continued pretraining","small language models","low-resource NLP","model evaluation","reproducibility","multilingual benchmarks","preference optimization"],"falsifier":"Replace the two machine-translated benchmarks with human-written Dutch equivalents and add a native-speaker fluency judgment; if Fietje Chat then falls behind GEITje 7B Ultra and Tweety, the paper's competitive claim is falsified.","tokens_in":50745,"feed_emoji":"🇳🇱","tokens_out":8113,"duration_ms":65500,"temperature":0.7,"pith_summary":"This paper claims that a 2.7-billion-parameter model, built by continue-pretraining Phi-2 on 28 billion filtered Dutch tokens, can reach competitive quality for Dutch despite its small size. The stakes are practical: Dutch remains an underrepresented language in large language models, and Fietje is a fully open recipe, with weights, data, and training and evaluation code all public. The paper's central empirical finding is that the chat variant outperforms larger Dutch-specific models like GEITje 7B Ultra and Tweety on two of the five zero-shot benchmarks (ARC reasoning and Global MMLU). It also finds that recent small multilingual models outperform older and larger Dutch-adapted models, so release date and multilingual pretraining matter more than parameter count.","feed_headline":"A 2.7B Dutch model beats larger Dutch rivals in two tasks","feed_subtitle":"Continue-pretrained on 28B Dutch tokens, Fietje Chat is competitive while keeping weights, data, and code open.","key_machinery":"The load-bearing object is the continued-pretraining pipeline: starting from the English-centric Phi-2, the model is trained on a curated 28-billion-token Dutch mixture of Wikipedia and heavily filtered web data, then supervised fine-tuned on Dutch instruction conversations, then aligned with Direct Preference Optimization on Dutch preference pairs. This is what carries the adaptation claim. The evaluation side is carried by a uniform zero-shot protocol with constrained label decoding and five repeated runs per model, which produces confidence intervals for the benchmark scores and is meant to make the comparisons reproducible.","core_discovery":"The central claim is that continued pretraining can make a small, English-centric model genuinely Dutch: Fietje takes Phi-2, a 2.7-billion-parameter model, and trains it further on 28 billion Dutch tokens drawn from filtered Wikipedia and web data. From that base, an instruct version is trained on synthetic Dutch conversations and a chat version is aligned with Direct Preference Optimization. In the paper's evaluation, Fietje Chat beats GEITje 7B Ultra and Tweety, both 7-billion-parameter models built specifically for Dutch, on ARC and Global MMLU, and the author states that upon its release Fietje was the best Dutch model in its weight class. The same evaluation shows that newer multilingual small models such as Phi 3.5 and Qwen 2.5 have since gone past all Dutch-specific models tested, which the paper reads as evidence that multilingual pretraining and release date now dominate size.","pith_inferences":["The paper's own limitation section implies that its label-choice benchmarks do not measure Dutch fluency; an open-ended generation test with native speakers would be a stronger test of the competitive claim.","The strict data filters used here probably remove some legitimate Dutch text, so a more nuanced quality classifier could unlock the larger 55-billion-token subset that the author also releases.","If continued pretraining is combined with vocabulary expansion, the resulting small model could plausibly close the remaining gap to large multilingual models on Dutch-specific tasks."],"forward_implications":["A 2.7-billion-parameter Dutch model can serve tasks that previously required 7B Dutch-specific models, lowering hardware and cost barriers for Dutch-language applications.","The public release of weights, data, and training and evaluation code makes Fietje a reusable template for adapting small LLMs to other underrepresented languages.","Future Dutch adaptation should start from current multilingual base models rather than older English-only ones, since newer small multilingual models already outperform older adapted 7B models.","Combining a Dutch-optimized tokenizer with continued pretraining is a likely next step, because tokenizer efficiency and corpus adaptation improve different aspects of Dutch processing.","Rankings built on translated benchmarks should be read with caution until Dutch-native benchmarks that also measure fluency are available."],"supporting_citations":[{"why":"Supplies the base Phi-2 model that Fietje continue-pretrains for Dutch.","marker":"Javaheripi et al. 2023"},{"why":"Supplies the large multilingual CulturaX corpus from which the Dutch pretraining data is drawn and filtered.","marker":"Nguyen et al. 2024"},{"why":"Supplies the synthetic Dutch instruction and preference datasets used for Fietje's instruct and chat stages, and the GEITje 7B Ultra baseline.","marker":"Vanroy 2024"},{"why":"Supplies GEITje 7B, a Dutch continued-pretrained Mistral model used as the main comparison point.","marker":"Rijgersberg and Lucassen 2023"},{"why":"Supplies Tweety, the tokenizer-updated Dutch 7B model that Fietje Chat outperforms on two benchmarks.","marker":"Remy et al. 2024"},{"why":"Provides the Direct Preference Optimization algorithm used to align Fietje Chat.","marker":"Rafailov et al. 2024"},{"why":"Provides the ARC reasoning benchmark that is one of the two tasks where Fietje Chat beats larger Dutch-specific models.","marker":"Clark et al. 2018"},{"why":"Provides Global MMLU, the world-knowledge benchmark that is the other task where Fietje Chat wins.","marker":"Singh et al. 2024"},{"why":"Provides DBRD, a native Dutch sentiment benchmark anchoring the evaluation in genuinely Dutch text.","marker":"van der Burgh and Verberne 2019"},{"why":"Provides Dutch CoLA, the grammatical-acceptability benchmark used to measure Dutch linguistic competence.","marker":"Bylinina et al. 2024"}],"fun_headline_variants":["2.7B Dutch model beats 7B rivals in ARC and Global MMLU","Fietje: open 2.7B model outperforms larger Dutch LLMs","Small open Dutch model tops 7B competitors in two benchmarks","Continue-pretrained 2.7B beats 7B Dutch models on reasoning","Dutch 2.7B LLM with open weights surpasses 7B models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the five zero-shot benchmarks—two machine-translated and all of them label-selection tasks—reflect real Dutch language quality; the paper itself notes it does not measure fluency, so if those benchmarks are biased the competitive ranking could change.","fun_headline_variants_meta":{"raw":{"variants":["2.7B Dutch model beats 7B rivals in ARC and Global MMLU","Fietje: open 2.7B model outperforms larger Dutch LLMs","Small open Dutch model tops 7B competitors in two benchmarks","Continue-pretrained 2.7B beats 7B Dutch models on reasoning","Dutch 2.7B LLM with open weights surpasses 7B models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2103,"prompt_tokens":943,"completion_tokens":1160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1053}},"tokens_in":559,"tokens_out":1160,"duration_ms":8144,"temperature":1.0,"reasoning_tokens":1053,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:25:00.834338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the two machine-translated benchmarks with human-written Dutch equivalents and add a native-speaker fluency judgment; if Fietje Chat then falls behind GEITje 7B Ultra and Tweety, the paper's competitive claim is falsified.","supporting_citations":[],"review_version":1}