Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

GeistBERT: Breathing Life into German NLP

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GeistBERT establishes the best results among base models on six German NLP benchmarks, including an absolute state of the art on GermEval 2018 fine-grained tweet classification (66.42% F1), by continuing pretraining from a GottBERT…

desk verdict Useful model release with honest limitations, but the headline SOTA claim rests on best-of-24 point estimates against borrowed baselines with no variance; the 9-point jump on GermEval-2018-fine needs a matched rerun. read the letter →

arxiv 2506.11903 v5 pith:K2YJVAH6 submitted 2025-06-13 cs.CL

Raphael Scheible-Schmitt , Johann Frei This is my paper
classification cs.CL
keywords GermanNLPpre-trainedlanguagemodelcontinuedpretrainingwholewordmaskingRoBERTatextclassificationnamedentityrecognitionnaturalinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GeistBERT is a German-language transformer that the authors pre-trained by continuing from an existing German RoBERTa model, GottBERT, rather than starting from scratch. They added roughly 1.3 TB of German text from web, parallel, Wikipedia, and legal sources, switched pre-training to whole-word masking, and kept the base model size at about 126 million parameters. Across six standard German NLP tasks, GeistBERT ranks first among all base models compared, sets an absolute state of the art on GermEval 2018 fine-grained sentiment classification (66.42% F1), and beats several models with two to four times more parameters. The paper's point is that a carefully chosen corpus and masking scheme can push a compact, openly released German model past much larger ones.

What carries the argument

The load-bearing mechanism is continued pretraining: GeistBERT starts from the f GottBERTbase checkpoint instead of random weights, so the model inherits German tokenization and representations and then sees a 1.3 TB mixture with Whole Word Masking. WWM masks all subword tokens belonging to a selected word together rather than a random subword, and under the RoBERTa base schedule with dynamic masking this is intended to force the model to learn word-level semantics. The paper also notes that GPU training allowed sentence-aware segmentation, keeping fixed 512-token sequences from cutting through sentences, and that warm-starting gave a much lower initial perplexity than training from scratch.

What would settle it

Fine-tune GeistBERT and the GottBERT baselines under the same grid but report the mean and standard deviation of all 24 runs instead of only the best validation checkpoint; if GeistBERT's mean does not exceed the baselines' means on GermEval 2018 fine and XNLI, the state-of-the-art claim is a selection artifact. Alternatively, pre-train a matched control from the same f GottBERTbase checkpoint on the same 1.3 TB corpus with standard subword masking and no added subcorpora; if its downstream scores match GeistBERT's, the recipe's components are not the cause.

Watch

Extended reading notes

Core claim

GeistBERT is a German RoBERTa-base model, about 126 million parameters, obtained by continued pretraining from the best filtered GottBERT checkpoint on roughly 1.3 TB of German text drawn from a cleaned multilingual web corpus (OSCAR23 and mC4 via CulturaX), several OPUS parallel corpora, German Wikipedia, and OpenLegalData. Pre-training uses RoBERTa's default base configuration, dynamic masking, and Whole Word Masking, with sequences kept inside natural sentence boundaries. On six German benchmarks, GeistBERT leads every base model compared, reaches an absolute state of the art on GermEval 2018 fine-grained classification (66.42% F1), and surpasses several large-scale models, mainly on classification; the paper attributes this to corpus scale and diversity plus whole-word masking.

Load-bearing premise

The load-bearing premise is that the improved scores come from the pretraining recipe—the larger, more varied corpus plus whole-word masking—rather than from choosing the best of 24 fine-tuning runs per model, since the paper reports no variance and performs no ablation of either component.

Editorial extensions

If this is right

  • GeistBERT leads every base model in the comparison on all six tasks, so it provides a drop-in stronger baseline for German NLP at the same parameter count.
  • It reaches the top score on GermEval 2018 fine-grained classification (66.42% F1) even among large models, showing that base-size German models can win specialized benchmarks.
  • On German NLI, GeistBERT (82.67% accuracy) surpasses the large GottBERT variant (82.46%) despite having roughly one-third the parameters, narrowing the practical gap to large models.
  • Continued pre-training from an existing checkpoint reaches usable representations quickly, with validation perplexity starting at 35.17 and the full run taking about 8.3 days on eight A40 GPUs, making the recipe reproducible at modest compute.
  • Releasing the model under the MIT license lets other teams adopt the new baseline directly in downstream tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the recipe's benefits are real, the same warm-start-plus-diverse-corpus-plus-whole-word-masking pattern could transfer to other mid-resource languages with existing monolingual checkpoints and cleaned web corpora, potentially producing similar base-model gains without full-from-scratch pretraining costs.
  • A morphology-probing evaluation would sharpen the causal story: whole-word masking should most help long German compounds and inflected forms, the tokens subword masking fragments; if gains concentrate there, the mechanism is supported, and if not, the corpus mix is the likelier driver.
  • Because the paper reports only the best of 24 fine-tuning runs per model and no variance, the published gaps are best-case figures; a distribution-based comparison could reorder the leaderboard even if the underlying model is genuinely strong.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents GeistBERT, a German RoBERTa-base model obtained by continued pretraining of the filtered GottBERT base checkpoint on a 1.3 TB corpus composed of CulturaX (OSCAR23 + mC4), several OPUS corpora, Wikipedia, and OpenLegalData. Pretraining uses fairseq with whole word masking, dynamic masking, and fixed 512-token sequences. The model is evaluated on German XNLI, CoNLL-2003, GermEval 2014, GermEval 2018 coarse and fine, and 10kGNAD, with all scores reported as the best of 24 fine-tuning runs selected on validation. The authors claim a new state of the art among base models for German NLP, an absolute state of the art on GermEval 2018 fine-grained classification, and competitive performance with large models. The comparison baselines, including all GottBERT numbers, are taken from the authors' previous study (Scheible et al., 2024). The paper also discusses training dynamics, data quality, and limitations, and releases the model under the MIT license.

Significance. If the reported results are robust, the contribution is valuable: GeistBERT is a publicly released, base-sized German model that appears to close much of the gap to large models on several benchmarks, and the continued-pretraining recipe is simple and reproducible. The authors deserve credit for documenting compute time, releasing checkpoints, and explicitly listing limitations. However, the central empirical claim rests on a comparison between GeistBERT scores produced by a best-of-24 validation-selection protocol and baseline scores borrowed from an earlier paper that may not have used the same protocol. No variance estimates, no matched reruns, and no ablations of whole word masking or corpus composition are provided. The largest claimed effect, a 10-13 point jump on GermEval 2018 fine, is anomalous and unexplained. For these reasons the paper is not yet convincing as a state-of-the-art claim, although the underlying model and evaluation scaffolding are plausible and worth publishing after controlled verification.

major comments (4)
  1. [Section 3.2, Table 6] Table 6 reports GeistBERT's scores as the best of 24 fine-tuning runs selected on validation, while every non-GeistBERT row is 'taken from Scheible et al. (2024).' The manuscript does not establish that those baseline numbers were produced under the same protocol: the same learning-rate and batch-size grid, the same 24-run selection rule, the same validation splits, and the same test-set versions. If the baselines were not selected the same way, the comparison is asymmetric and the headline state-of-the-art claim is not supported. I ask for a controlled rerun of at least fGottBERTbase, GottBERTbase, GBERTbase, and GELECTRAbase under the Section 3.2 protocol, with the mean and standard deviation over seeds, and a statement of whether any baseline improves when re-run.
  2. [Section 4.2, Table 6] The GermEval 2018 fine-grained result is anomalous: GeistBERT scores 66.42 versus 52.82 for fGottBERTbase and 56.17 for the best large model, a 10-13 point gap, whereas other tasks show 1-3 point gaps. No explanation is offered, and the paper does not rule out that the gain is an artifact of checkpoint selection, a different validation split, or leakage from the 1.3 TB web corpus into the GermEval 2018 test set. Please report per-class F1, the number of overlapping tweets between the training data and the test set, and the distribution of the 24 validation F1 scores for GeistBERT and at least fGottBERTbase.
  3. [Section 5.2 and Limitations] The paper's central explanatory claim, that whole word masking and corpus diversity drove the improvements, is not tested. Section 5.2 explicitly concedes that no WWM ablation and no per-subcorpus ablation were performed, and the Limitations section repeats this. Consequently, the 'clear improvements' attributed to these factors in Sections 5.1 and 6 are not supported by controlled evidence; the end-to-end comparison alone cannot separate the effects of the new corpus, WWM, sentence-aware segmentation, the higher peak learning rate, and continued pretraining. I ask either for ablations or for removal or substantial rephrasing of these causal attributions.
  4. [Section 4.1, Figure 1] The perplexity trajectory is at odds with the claimed improvement. GeistBERT's final perplexity is approximately 11, while fGottBERTbase converged to approximately 4; the text notes the initial values but does not discuss why the continued-pretrained model ends with higher held-out perplexity. If the pretraining validation set is comparable, this suggests the model has not improved language modeling under its own metric, and the downstream gains would need to be explained by task-specific effects or by a distribution shift in the perplexity evaluation set. Please clarify the evaluation set used for Figure 1, define 'optimization cycle,' and discuss this discrepancy explicitly.
minor comments (6)
  1. [Table 5] Table 5 lists two rows for GottBERTbase and two for fGottBERTbase, but the caption's notation for the last checkpoint (the symbol after 'last' in the caption) appears to be missing from the rendered table, making the row labels ambiguous.
  2. [Table 2 and Section 3.2] Table 2 lists 'Epochs: 30' as a single value, while Section 3.2 states that NLI tasks ran for up to 10 epochs using fairseq-adapted hyperparameters; clarify the epoch budget for NLI in the table or in the text.
  3. [Section 3.2] The sentence 'Each task was executed 24 times with varied hyperparameters' should say '24 hyperparameter combinations' because Table 2 defines a 6-by-4 grid of learning rates and batch sizes; the current wording implies repeated seeds, which the paper does not report.
  4. [Figure 1] Figure 1 does not specify what validation set is used for perplexity or how 'optimization cycle' is defined; add axis labels, a definition of the evaluation set, and a note on whether the perplexity values are computed on the same held-out data for both models.
  5. [Section 5.2] The claim that 'WWM contributed to improved tokenization' is not supported by any reported tokenization statistic; either report a tokenization-level comparison or soften the claim to a hypothesis.
  6. [Abstract and Section 1] The abstract describes 'incrementally training on a diverse corpus' without mentioning that the model is initialized from fGottBERTbase; because continued pretraining is a key design choice, it should be stated in the abstract.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports externally measured benchmark results; missing ablations and best-of-24 selection are robustness concerns, not circular reasoning.

full rationale

GeistBERT is an empirical model paper with no mathematical derivation chain in which an output is forced by its inputs. The central claim—new state of the art among German base models—rests on fine-tuned test-set scores on standard external benchmarks (XNLI, CoNLL-2003, GermEval 2014/2018, 10kGNAD), with GeistBERT's own scores measured in this study. The comparison values for non-GeistBERT models are taken from the same authors' prior GottBERT paper, but those are externally falsifiable benchmark results, not quantities derived from GeistBERT or fitted to the claim. Continued pretraining from f GottBERTbase could make improvement unsurprising, but the observed superiority is an empirical outcome, not an artifact of definition. The paper explicitly concedes that no WWM ablation and no per-subcorpus ablation were run; attributing gains to WWM and corpus diversity is an interpretation supported by external citations rather than a circular derivation. The best-of-24 checkpoint selection and absence of variance estimates are legitimate statistical robustness concerns but do not constitute circularity by construction. No quoted step exhibits an equation or a fitted parameter being renamed as a prediction, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

All free parameters are optimization choices; no new theoretical entities are introduced. The main unproven assumptions are causal: that the corpus and WWM, not evaluation selection, produced the gains.

free parameters (3)
  • Downstream learning rate (per task and model, from grid search) = 1e-6 to 5e-5, e.g., GeistBERT uses 2e-5 for CoNLL-03 and 1e-6 for 10kGNAD (Table 5)
    The best LR per task and model is selected by validation performance; this is a fitted hyperparameter and part of why GeistBERT's scores are best-of-24.
  • Downstream batch size (per task and model, from grid search) = 16, 32, 48, or 64 depending on task and model (Table 5)
    Also selected by validation performance; affects optimization and final metrics.
  • Pre-training peak learning rate, warmup steps, batch size, and update steps = peak LR 0.0007; warmup 10k; batch 8k; steps 100k
    Chosen by hand following RoBERTa defaults with an elevated LR; not tuned in this paper, but still a modeling choice that affects the trained model.
assumptions (4)
  • domain assumption Continued pretraining on a larger, more diverse corpus improves downstream performance over the starting checkpoint.
    The paper's main contribution is continued pretraining; this premise is supported by prior literature but not tested against a from-scratch training run or a same-corpus ablation.
  • ad hoc to paper Whole Word Masking improves German tokenization and downstream performance.
    The paper integrates WWM but explicitly states no ablation was performed (Section 5.2); still, it attributes improvements partly to WWM.
  • ad hoc to paper Choosing the best validation checkpoint out of 24 fine-tuning runs gives a reliable estimate of test performance.
    Not tested; selection on validation can still inflate test scores when many runs are tried, and no error bars are reported.
  • domain assumption The comparison numbers from Scheible et al. (2024) are directly comparable because the evaluation protocol was copied.
    All non-GeistBERT scores in Table 6 are taken from that prior study and not re-run under identical compute or code versions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GeistBERT: Breathing Life into German NLP." pith.science (2026). https://pith.science/paper/K2YJVAH6

@misc{pith2026250611903,
  author       = {Pith},
  title        = {Pith review of: GeistBERT: Breathing Life into German NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K2YJVAH6}},
  note         = {Machine review of arXiv:2506.11903}
}
abstract

Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain from updated architectures and modern datasets tailored to the linguistic characteristics of the German language. GeistBERT seeks to improve German language processing by incrementally training on a diverse corpus and optimizing model performance across various NLP tasks. We pre-trained GeistBERT using fairseq, following the RoBERTa base configuration with Whole Word Masking (WWM), and initialized from GottBERT weights. The model was trained on a 1.3 TB German corpus with dynamic masking and a fixed sequence length of 512 tokens. For evaluation, we fine-tuned the model on standard downstream tasks, including NER (CoNLL 2003, GermEval 2014), text classification (GermEval 2018 coarse/fine, 10kGNAD), and NLI (German XNLI), using $F_1$ score and accuracy as evaluation metrics. GeistBERT achieved strong results across all tasks, leading among base models and setting a new state-of-the-art (SOTA) in GermEval 2018 fine text classification. It also outperformed several larger models, particularly in classification benchmarks. To support research in German NLP, we release GeistBERT under the MIT license.

Figures

Figures reproduced from arXiv: 2506.11903 by the authors.

Figure 1
Figure 1. Perplexity of fGottBERTbase and Geist￾BERT, evaluated on the validation set after each op￾timization cycle; values are plotted on a logarithmic y-axis. 4.2 Downstream Tasks GeistBERT sets a new state-of-the-art among base models for German NLP, outperforming all compa￾rable models and closely approaching large-scale model performance across tasks. It even achieves absolute SOTA in GermEval 2018 fine-grained clas￾sif… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HalleluBERT: Let Every Token That Has Meaning Bear Its Weight

    cs.CL 2025-10 conditional novelty 5.0 of 10

    HalleluBERT, a Hebrew-only RoBERTa encoder family trained from scratch at scale, reports the highest unweighted mean scores on BMC, NEMO, and SMCD benchmarks, but without statistical significance testing.

  2. SindBERT, the Sailor: Charting the Seas of Turkish NLP

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SindBERT releases Turkish RoBERTa base/large models trained on 312GB of text; they match existing models, with the large variant best on two of four tasks and little scaling gain.

Reference graph

Works this paper leans on

34 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Veronika Arefeva and Roman Egger. 2022. https://doi.org/10.3390/digital2040030 When bert started traveling: Tourbert—a natural language processing model for the travel industry . Digital, 2(4):546--559

    work page doi:10.3390/digital2040030 2022

  4. [4]

    Darina Benikova, Chris Biemann, Max Kisselew, and Sebastian Padó. 2014. GermEval 2014 Named Entity Recognition Shared Task : Companion Paper . Proceedings of the KONVENS GermEval Shared Task on Named Entity Recognition, pages 104--112

    work page 2014

  5. [5]

    Bressem, Jens-Michalis Papaioannou, Paul Grundmann, Florian Borchert, Lisa C

    Keno K. Bressem, Jens-Michalis Papaioannou, Paul Grundmann, Florian Borchert, Lisa C. Adams, Leonhard Liu, Felix Busch, Lina Xu, Jan P. Loyen, Stefan M. Niehues, Moritz Augustin, Lennart Grosser, Marcus R. Makowski, Hugo J.W.L. Aerts, and Alexander Löser. 2024. https://doi.org/https://doi.org/10.1016/j.eswa.2023.121598 medbert.de: A comprehensive german b...

    arXiv 2024

  6. [6]

    Branden Chan, Stefan Schweter, and Timo M \"o ller. 2020. https://www.aclweb.org/anthology/2020.coling-main.598 G erman ' s next language model . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788--6796, Barcelona, Spain (Online). International Committee on Computational Linguistics

    work page 2020

  7. [7]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475--2485, Brussels, Belgium. Association...

    doi:10.18653/v1/d18-1269 2018

  8. [8]

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. https://doi.org/10.1109/TASLP.2021.3124365 Pre-training with whole word masking for chinese bert . IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3504–3514

    arXiv 2021

Show all 34 references
  1. [9]

    Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.292 R ob BERT : a D utch R o BERT a-based L anguage M odel . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255--3265, Online. Associa...

    2020 doi

  2. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Associat...

    2019 doi

  3. [11]

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. http://arxiv.org/abs/2002.06305 Fine- Tuning Pretrained Language Models : Weight Initializations , Data Orders , and Early Stopping . arXiv:2002.06305 [cs]. ArXiv: 2002.06305

    2020 arXiv

  4. [12]

    Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, and Pierre Zweigenbaum. 2022. https://aclanthology.org/2022.lrec-1.281/ Re-train or train from scratch? comparing pre-training strategies of BERT in the medical domain . In Proceedings of the Thirteenth Language Resources a...

    2022

  5. [13]

    Johann Frei, Ludwig Frei-Stuber, and Frank Kramer. 2022. http://arxiv.org/abs/2206.14504 Gernermed++: Transfer learning in german medical nlp

    2022 arXiv

  6. [14]

    Johann Frei and Frank Kramer. 2023. https://doi.org/https://doi.org/10.1016/j.jbi.2023.104478 Annotated dataset creation through large language models for non-english medical nlp . Journal of Biomedical Informatics, 145:104478

    2023

  7. [15]

    Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. http://arxiv.org/abs/2004.10964 Don't stop pretraining: Adapt language models to domains and tasks . CoRR, abs/2004.10964

    2020 arXiv

  8. [16]

    Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. 2022. http://arxiv.org/abs/2212.10440 Perplexed by quality: A perplexity-based method for adult and harmful content detection in multilingual heterogeneous web data

    2022 arXiv

  9. [17]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. https://doi.org/10.1093/bioinformatics/btz682 Biobert: a pre-trained biomedical language representation model for biomedical text mining . Bioinformatics, 36(4):1234--1240

    2019 doi

  10. [18]

    Manuel Lentzen, Sumit Madan, Vanessa Lage-Rupprecht, Lisa Kühnel, Juliane Fluck, Marc Jacobs, Mirja Mittermaier, Martin Witzenrath, Peter Brunecker, Martin Hofmann-Apitius, Joachim Weber, and Holger Fröhlich. 2022. https://doi.org/10.1093/jamiaopen/ooac087 Critical assessment ...

    2022 doi

  11. [19]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 RoBERTa : A Robustly Optimized BERT Pretraining Approach . arXiv:1907.11692 [cs]. ArXiv: 1907.11692

    2019 arXiv

  12. [20]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, \' E ric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. https://www.aclweb.org/anthology/2020.acl-main.645 CamemBERT : a Tasty French Language Model . In Proceedings of the 58th Ann...

    2020

  13. [21]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. http://arxiv.org/abs/2309.09400 Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages

    2023 arXiv

  14. [22]

    Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. https://www.aclweb.org/anthology/2020.acl-main.156 A Monolingual Approach to Contextualized Word Embeddings for Mid - Resource Languages . In Proceedings of the 58th Annual Meeting of the Association for Comput...

    2020

  15. [23]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. http://arxiv.org/abs/1904.01038 fairseq: A Fast , Extensible Toolkit for Sequence Modeling . arXiv:1904.01038 [cs]. ArXiv: 1904.01038

    2019 arXiv

  16. [24]

    Julian Risch, Eva Krebs, Alexander Löser, Alexander Riese, and Ralf Krestel. 2018. Fine- Grained Classification of Offensive Language . In Proceedings of GermEval 2018 (co-located with KONVENS ) , pages 38--44

    2018

  17. [25]

    Dietmar Schabus, Marcin Skowron, and Martin Trapp. 2017. https://doi.org/10.1145/3077136.3080711 One Million Posts : A Data Set of German Online Discussions . In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval ( S...

    2017

  18. [26]

    Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, and Martin Boeker. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1183 G ott BERT : a pure G erman language model . In Proceedings of the 2024 Conference ...

    2024 doi

  19. [27]

    Moritz Scherrmann. 2023. http://arxiv.org/abs/2311.08793 German finbert: A german pre-trained language model

    2023 arXiv

  20. [28]

    J \"o rg Tiedemann. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf Parallel data, tools and interfaces in OPUS . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC '12) , pages 2214--2218, Istanbul, Turkey. ...

    2012

  21. [29]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. 2003. https://doi.org/10.3115/1119176.1119195 Introduction to the CoNLL -2003 Shared Task : Language - Independent Named Entity Recognition . In Proceedings of the Seventh Conference on Natural Language Learning at HLT - NAACL 2003 -...

    2003

  22. [30]

    Cagri Toraman, Eyup Halit Yilmaz, S ahinu c Furkan, and Oguzhan Ozcelik. 2023. https://doi.org/10.1145/3578707 Impact of tokenization on language models: An analysis for turkish . ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4)

    2023 doi

  23. [31]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  24. [32]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \' e mi Louf, Morgan Funtowicz, and Jamie Brew. 2019. http://arxiv.org/abs/1910.03771 Huggingface's transformers: State-of-the-art natural language processing...

    2019 arXiv

  25. [33]

    Haoran Xu, Benjamin Van Durme, and Kenton Murray. 2021. http://arxiv.org/abs/2109.04588 Bert, mbert, or bibert? a study on contextualized embeddings for neural machine translation

    2021 arXiv

  26. [34]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.