Pith. sign in

REVIEW 4 cited by

BERTje: A Dutch BERT Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1912.09582 v1 pith:CQNJWC4X submitted 2019-12-19 cs.CL

Wietse de Vries , Andreas van Cranenburgh , Arianna Bisazza , Tommaso Caselli , Gertjan van Noord , Malvina Nissim This is my paper
classification cs.CL
keywords bertmodelbertjedutchlanguagemultilingualpre-trainedtasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch

    cs.CL 2025-07 conditional novelty 6.0 of 10

    The paper presents a Dutch adaptation of the CrowS-Pairs bias benchmark and reports bias scores for seven masked and two autoregressive language models across nine demographic categories.

  2. FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...

  3. Assessing the Role of Data Quality in Training Bilingual Language Models

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.

  4. Towards Fairness Assessment of Dutch Hate Speech Detection

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Counterfactual data generation and fine-tuning improve group fairness metrics for Dutch hate speech detection while slightly hurting counterfactual token fairness on toxic content.

Pith tools