A three-stage continual training recipe (tokenizer change, embedding alignment, full retraining) produces NorMistral-11B, an open Norwegian and Northern Sámi language model that improves on most Norwegian benchmarks and is 30% faster at inference.
GPT or BERT: why not both?
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Small Languages, Big Models: A Study of Continual Training on Languages of Norway
A three-stage continual training recipe (tokenizer change, embedding alignment, full retraining) produces NorMistral-11B, an open Norwegian and Northern Sámi language model that improves on most Norwegian benchmarks and is 30% faster at inference.