REVIEW 10 cited by
BERTje: A Dutch BERT Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.
Forward citations
Cited by 10 Pith papers
-
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
A Dutch governmental LLM benchmark suite reveals consistent quality-cost-energy trade-offs and a dissociation between factuality and honesty across 31 models.
-
Reading Between the Signs: Predicting Future Suicidal Ideation from Adolescent Social Media Texts
A model trained on earlier, non-suicidal forum interactions predicts whether a user will later write a post about suicidal thoughts, reaching 0.73 balanced accuracy on a Dutch youth forum.
-
Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch
The paper presents a Dutch adaptation of the CrowS-Pairs bias benchmark and reports bias scores for seven masked and two autoregressive language models across nine demographic categories.
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...
-
Multi-perspective Alignment for Increasing Naturalness in Neural Machine Translation
A reinforcement learning method that rewards both naturalness and content preservation improves lexical diversity in English-to-Dutch literary machine translation, with mixed results on translation accuracy.
-
Training Bilingual LMs with Data Constraints in the Targeted Language
Higher-quality auxiliary English pretraining data improves target-language performance for languages close to English (about 2% on translated QA tasks), but not for distant languages, when target-language data is limi...
-
Assessing the Role of Data Quality in Training Bilingual Language Models
A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.
-
Towards Fairness Assessment of Dutch Hate Speech Detection
Counterfactual data generation and fine-tuning improve group fairness metrics for Dutch hate speech detection while slightly hurting counterfactual token fairness on toxic content.
-
ChocoLlama: Lessons Learned From Teaching Llamas Dutch
Continued pretraining with LoRA and a Dutch-specific tokenizer improves Llama-2's Dutch, but gives limited gains for already-multilingual Llama-3.
-
Can bidirectional encoder become the ultimate winner for downstream applications of foundation models?
A review of bidirectional encoder models (BERT and variants) and their performance on GLUE and SQuAD relative to one-way generative models.
Discussion (0). Continue with ORCID to comment.