Pith. sign in

REVIEW 4 major objections 6 minor 33 references

SindBERT gives Turkish NLP its first large-scale RoBERTa encoder, trained from scratch on 312 GB of text, and the benchmarks show scaling has stalled.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SindBERT releases Turkish RoBERTa base/large models trained on 312GB of text; they match existing models, with the large variant best on two of four tasks and little scaling gain.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The SindBERT release is a genuinely useful Turkish NLP resource; the benchmark claims are plausible but under-supported by missing variance analysis. the 4 major comments →

arxiv 2510.21364 v2 pith:WNAEULL3 submitted 2025-10-24 cs.CL

SindBERT, the Sailor: Charting the Seas of Turkish NLP

Raphael Schmitt , Stefan Schweter This is my paper
classification cs.CL
keywords Turkish NLPRoBERTaencoder pre-trainingcorpus curationscalinglinguistic acceptabilitypart-of-speech taggingnamed entity recognition
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to fill a gap: Turkish, a morphologically rich language, lacked a large-scale encoder-only model in the RoBERTa family trained from scratch. The authors pre-train SindBERT base and large on 312 GB of Turkish web text, release both under the MIT license, and evaluate on part-of-speech tagging, named entity recognition, offensive language detection, and the TurBLiMP acceptability benchmark. Their central finding is that SindBERT competes with the strongest Turkish and multilingual encoders, with the large variant best on two of four tasks, but scaling from base to large yields no consistent gains—a flat trend they also observe for two multilingual families. They argue this indicates current Turkish benchmarks are nearing saturation, and that corpus quality and diversity (as in the smaller, curated BERTurk) can matter more than raw data volume. A sympathetic reader would care because the paper both provides an openly usable resource and reframes where progress in Turkish NLP should come from.

Core claim

The paper claims that a from-scratch RoBERTa encoder for Turkish, trained on 312 GB of lightly filtered web text, reaches competitive performance across POS tagging, NER, offensive language detection, and TurBLiMP, with the large variant achieving the best scores in two of four tasks. Scaling does not produce uniform gains; the flat trend observed for SindBERT, XLM-R, and EuroBERT suggests many Turkish benchmarks are already saturated. Comparing SindBERT with the smaller but cleaner BERTurk shows that corpus curation can outweigh volume for linguistically sensitive evaluations. The models are released openly under the MIT license in both base and large configurations.

What carries the argument

The central object is SindBERT, a pair of RoBERTa-style encoders (base, 126M parameters; large, 357M) trained from scratch on 312 GB of Turkish text using a purpose-built 52k byte-level BPE vocabulary. The machinery is a standard RoBERTa pretraining regime—100k updates, global batch size 8k, full precision—combined with a four-task evaluation suite and a size-versus-curation contrast against existing Turkish and multilingual encoders. The vocabulary design and the comparison against BERTurk carry much of the argument about tokenization and data quality.

Load-bearing premise

The ranking and scaling conclusions rest on the assumption that single-run fine-tuning scores, each chosen as the best of ten hyperparameter configurations from one fixed seed per model size, are stable enough to order models that differ by only 0.1–0.5 points; the paper itself acknowledges that seed-related variance may cause minor deviations.

What would settle it

Re-run the fine-tuning for at least the base models with 10 random seeds per configuration and report mean and standard error on POS, NER, and OffensEval-TR. If the 0.1–0.5 point gaps between SindBERT and the best Turkish encoders shrink below noise (overlapping error bars), the specific 'best in two tasks' and 'flat scaling' claims are not established, even though the resource release remains valid.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Turkish NLP gains the first openly released, MIT-licensed large-scale RoBERTa encoder, usable in both base and large configurations.
  • If the saturation claim holds, future gains for Turkish should come from harder benchmarks, curated corpora, and adaptive tokenization rather than parameter growth.
  • The flat scaling observed across three model families means existing Turkish benchmarks may not discriminate model quality well; new evaluation suites are needed.
  • SindBERT's 52k BPE vocabulary, smaller than BERTurk's 128k variant, still achieves competitive results, suggesting vocabulary size interacts with corpus noise and morphology.
  • The comparison with BERTurk indicates that for morphologically rich but moderately resourced languages, data curation may dominate scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the saturation claim would be running the same models on TrGLUE or a longer-context benchmark; if scaling gaps reappear there, saturation is task-specific, not a general property of Turkish.
  • The single-run, fixed-seed evaluation with no error bars means the 0.1–0.5 point differences that separate SindBERT from top base models could shift with seeds; multi-seed reruns would sharpen or weaken the ranking claims.
  • The paper notes no deduplication was applied; since mC4 is known to contain near-duplicate web pages, the effective corpus may be smaller and noisier than 312 GB suggests, which would reinforce the corpus-quality conclusion.
  • SindBERTlarge's severe drop on ellipsis (27.8) versus SindBERTbase (59.0) suggests added capacity can hurt on rare syntactic phenomena—a testable hypothesis about large encoders and compositional generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SindBERT, a pair of RoBERTa-based encoder language models (base and large) pre-trained from scratch on 312 GB of Turkish text drawn from mC4, OSCAR23, and Wikipedia, with a custom 52k-byte-level BPE vocabulary. The models are released under the MIT license in fairseq and Hugging Face formats. The authors evaluate the models on part-of-speech tagging, WikiANN NER, OffensEval-TR offensive language detection, and the TURBLIMP linguistic acceptability benchmark, comparing against a range of Turkish and multilingual encoders. The paper claims that SindBERT is the first large-scale RoBERTa encoder for Turkish, that SindBERTlarge achieves the best results in two of four tasks, and that flat scaling trends across SindBERT, XLM-R, and EuroBERT indicate benchmark saturation, with corpus quality outweighing raw data volume.

Significance. The resource contribution is potentially valuable: open, MIT-licensed base and large RoBERTa encoders for Turkish, with detailed pre-training configuration, evaluation protocol, and compute reporting, would be a useful addition for Turkish NLP. The TURBLIMP evaluation, in particular, provides a linguistic-diagnostic perspective that is relatively rare in resource papers. If the comparative claims were established, the flat-scaling and corpus-curation conclusions would also be a meaningful data point for the field. However, the empirical claims are currently not supported by the reported evidence: the headline results rest on single fixed-seed runs with no variance estimates, and several margins are smaller than plausible fine-tuning noise. The resource release can stand on its own, but the scientific conclusions require additional evidence or explicit softening.

major comments (4)
  1. [§3.5 and Tables 4-5] The central comparative claims rest on runs with a single fixed seed per scale: seed 1 for all base models and seed 42 for all large models. There is no variance information, confidence interval, or significance test. The wording in §3.5 is internally inconsistent: it says 'reported scores are averaged across seeds for the best configuration' after stating that each model used a fixed seed. The decisive margins are small — SindBERTlarge edges XLM-Rlarge by 0.24 micro-F1 on PoS (94.63 vs. 94.39) and by 0.30 macro-F1 on OffensEval (82.29 vs. 81.99). Without multi-seed runs, the claim that SindBERTlarge 'achieves the best results in two tasks' is not established.
  2. [§5.1 and §5.3] The flat-scaling conclusion is load-bearing and is confounded with seed assignment. Because all base models use seed 1 and all large models use seed 42, every cross-scale comparison — including SindBERTbase vs. SindBERTlarge and XLM-Rbase vs. XLM-Rlarge — potentially reflects a seed effect rather than a scaling effect. The paper cites Dodge et al. (2020) for seed sensitivity but does not apply that knowledge to its own protocol. At minimum, re-run the large models with seed 1 (or all models with multiple seeds) for the tasks that drive the scaling claims, or reframe the scaling statements as observations on single runs.
  3. [Limitations and Table 4] The paper itself acknowledges that ConvBERTurk and ConvBERTurkmC4 may have an ambiguous generator/discriminator conversion and that this 'explains the suboptimal performance on TURBLIMP.' Yet ConvBERTurk ranks as the top or near-top base model on PoS and OffensEval, and these results are used to support the conclusion that corpus quality and architecture choices matter more than scale. If the ConvBERT checkpoints are unreliable, some of the comparative rankings and the 'curated corpus' argument are partially built on an unverified baseline. The authors should either verify the conversion and re-run, or clearly separate the affected conclusions from the verified model comparisons.
  4. [Tables 4-5 and §3.5] There is no indication of how stable the test-set numbers are under hyperparameter selection. The protocol selects the best of 10 validation hyperparameter configurations and then reports a single test-set number; this selection can overfit the validation signal, and without repeated trials there is no estimate of the variance introduced by early stopping and fine-tuning seed. I recommend reporting error bars across at least 3–5 seeds for the reported metrics, or explicitly labeling all results as single-run and removing the word 'best' from the conclusion when the gap is smaller than the expected noise.
minor comments (6)
  1. [Table 4] The fourth column is labeled 'Sentiment' although the task is OffensEval-TR 2020 offensive language detection; use a consistent label with §3.4 and Table 7.
  2. [§1] The sentence 'The only RoBERTa models out there were not computed in its fullest extend' should be rewritten for grammar and clarity.
  3. [Table 1] The RoBERTurk row reads 'OSCAR, Turkish C4 subset (1 GB) 28 GB' and is ambiguous about whether the total corpus size is 28 GB and the C4 subset is 1 GB, or something else. Clarify the corpus composition and size.
  4. [Related Work / References] XLM-R is cited as 'Chan, 2020' (a towardsdatascience blog). Cite the original XLM-R paper (Conneau et al., 2020) instead.
  5. [§3.3] The claim that the dataset size 'only permitted roughly four epochs' would be more informative if the total token count corresponding to 100k steps at batch size 8k were reported, since that is what actually determines the epoch count.
  6. [Table 5] The abbreviation 'A VG' in the last column is never expanded in the caption; define it as 'average overall TURBLIMP score'.

Circularity Check

0 steps flagged

No circular derivation; SindBERT's claims are empirical comparisons against external benchmarks, not reductions to their own inputs.

full rationale

The paper contains no derivation chain in which a predicted quantity is defined in terms of the same quantity or in which a fitted parameter is relabeled as a prediction. SindBERT is pre-trained from scratch on an openly described corpus (312 GB of mC4, OSCAR23, Wikipedia; §3.1) and then evaluated on four external benchmarks (UD PoS, WikiANN, OffensEval-TR, TurBLiMP) with pre-existing published datasets and protocols (§3.4), including direct comparisons against external models. The 'best in two tasks' and 'flat scaling' claims are empirical outcomes of those comparisons, not consequences of how SindBERT's training objective is set up. Self-citations to GottBERT and GeistBERT (§1, §2, §3.2, §3.5) supply architectural lineage and hyperparameter priors; they are not invoked to establish SindBERT's downstream scores or to rule out alternative explanations, so they are not load-bearing in the circularity sense. The fixed-seed protocol and missing variance estimates in §3.5 are a legitimate statistical-reliability concern for the small F1 margins in Tables 4–5, and the paper itself concedes 'minor deviations may still arise from seed-related variance' and the Limitations section notes 'baseline reproducibility introduces some uncertainty'; however, uncertainty about whether a lead is significant is not the same as the claimed result being equivalent to its input. The resource release and comparative benchmarks stand independently of any fitted relation, so no circular step is present.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new theoretical entities are introduced. The load-bearing choices are the training schedule, tokenizer size, and per-task hyperparameters selected on validation, plus the assumptions that single runs and a four-task suite support the saturation narrative.

free parameters (3)
  • Fine-tuning learning rate and batch size per model/task = varies; Table 7 (e.g., SindBERTbase PoS: LR 1e-5, BS 16; SindBERTlarge PoS: LR 1e-5, BS 32)
    Selected as the best of 10 grid-search combinations on validation for each of 15 models x 3 tasks; the reported test scores are those of the best validation config, so the comparison depends on these choices.
  • Pre-training schedule (100k steps, global batch 8k, 10k warmup, peak LR 4e-4 base / 1.5e-4 large) = 100k steps / batch 8k / LR as above
    Copied from GottBERT/RoBERTa and chosen by hand; no ablation in this paper justifies them for Turkish.
  • BPE vocabulary size 52,009 = 52,009
    Chosen by hand following GottBERT from a 40GB sample; the paper says it did not evaluate the effect of this choice (§3.2).
axioms (5)
  • domain assumption A single fine-tuning run per seed is sufficient to rank models
    Section 3.5 fixes seed 1 for base and 42 for large and Tables 4-5 report one score per cell; the rankings and 'best' claims depend on this assumption, which the authors themselves qualify with 'minor deviations may still arise from seed-related variance'.
  • ad hoc to paper The four benchmarks (PoS, WikiANN, OffensEval-TR, TURBLIMP) are representative enough to infer general Turkish benchmark saturation
    Section 5.1-5.2 generalizes from four tasks to 'current Turkish benchmarks may already be saturated'; the limitation section admits these do not cover QA, NLI, summarization, or long-context.
  • domain assumption Byte-level BPE trained on a 40GB sample is an adequate tokenizer for a morphologically rich language like Turkish
    Section 3.2 relies on prior Dutch/German experience and does not validate tokenizer choice for Turkish (the paper explicitly says it did not evaluate the effect).
  • domain assumption Lightly filtered, non-deduplicated web corpora are acceptable pretraining data for the claims made
    Section 3.1 and Limitations state only invalid-character filtering and no deduplication; if corpus noise severely hurt SindBERT, the comparisons against BERTurk would partly reflect data curation, not model architecture.
  • domain assumption The BLiMP scoring protocol transfers to Turkish minimal pairs
    Section 3.4 scores models by assigning higher probability to grammatical sentences; this assumes the probability ranking of the MLM is a valid acceptability measure for Turkish.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SindBERT, the Sailor: Charting the Seas of Turkish NLP." pith.science (2026). https://pith.science/paper/WNAEULL3

@misc{pith2026251021364,
  author       = {Pith},
  title        = {Pith review of: SindBERT, the Sailor: Charting the Seas of Turkish NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNAEULL3}},
  note         = {Machine review of arXiv:2510.21364}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Transformer models have revolutionized NLP, yet many morphologically rich languages remain underrepresented in large-scale pre-training efforts. With SindBERT, we set out to chart the seas of Turkish NLP, providing the first large-scale RoBERTa-based encoder for Turkish. Trained from scratch on 312~GB of Turkish text (mC4, OSCAR23, Wikipedia), SindBERT is released in both base and large configurations, representing the first large-scale encoder-only language model available for Turkish. We evaluate SindBERT on part-of-speech tagging, named entity recognition, offensive language detection, and the TurBLiMP linguistic acceptability benchmark. Our results show that SindBERT performs competitively with existing Turkish and multilingual models, with the large variant achieving the best scores in two of four tasks but showing no consistent scaling advantage overall. This flat scaling trend, also observed for XLM-R and EuroBERT, suggests that current Turkish benchmarks may already be saturated. At the same time, comparisons with smaller but more curated models such as BERTurk highlight that corpus quality and diversity can outweigh sheer data volume. Taken together, SindBERT contributes both as an openly released resource for Turkish NLP and as an empirical case study on the limits of scaling and the central role of corpus composition in morphologically rich languages. The SindBERT models are released under the MIT license and made available in both fairseq and Huggingface formats.

Figures

Figures reproduced from arXiv: 2510.21364 by Raphael Schmitt, Stefan Schweter.

Figure 1
Figure 1. Figure 1: Perplexity of the SindBERT models. Top: validation perplexity measured at checkpoints. Bottom: training perplexity measured at each optimization step. 4.2 Downstream Tasks Part-of-Speech Tagging Across base-scale mod￾els, performance on the Turkish Universal De￾pendencies treebank is consistently high, with micro-F1 values exceeding 93% for nearly all en￾coders. The strongest overall results are achieved b… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 2 canonical work pages

  1. [1]

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. https://doi.org/10.18653/v1/N19-4010 FLAIR : An Easy -to- Use Framework for State -of-the- Art NLP . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics ( Demonstrations ) , pages 54--59, Min...

    doi:10.18653/v1/n19-4010 2019

  2. [2]

    Ezgi Başar, Francesca Padovani, Jaap Jumelet, and Arianna Bisazza. 2025. https://arxiv.org/abs/2506.13487 Turblimp: A turkish benchmark of linguistic minimal pairs . Preprint, arXiv:2506.13487

    arXiv 2025

  3. [3]

    Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. 2025. https://arxiv.org/abs/2503.05...

    Pith/arXiv arXiv 2025

  4. [4]

    C a g r C \" o ltekin. 2020. https://www.aclweb.org/anthology/2020.lrec-1.758 A corpus of turkish offensive language on social media . In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6174--6184, Marseille, France

    2020

  5. [5]

    Branden Chan. 2020. https://towardsdatascience.com/xlm-roberta-the-multilingual-alternative-for-non-english-nlp-cf0b889ccbbf XLM - RoBERTa : The multilingual alternative for non-english NLP . Library Catalog: towardsdatascience.com

    2020

  6. [6]

    Branden Chan, Stefan Schweter, and Timo M \"o ller. 2020. https://doi.org/10.18653/v1/2020.coling-main.598 G erman ' s next language model . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788--6796, Barcelona, Spain (Online). International Committee on Computational Linguistics

    doi:10.18653/v1/2020.coling-main.598 2020

  7. [7]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://arxiv.org/abs/2003.10555 Electra: Pre-training text encoders as discriminators rather than generators . Preprint, arXiv:2003.10555

    Pith/arXiv arXiv 2020

  8. [8]

    Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020 a . http://arxiv.org/abs/2001.06286 RobBERT : a Dutch RoBERTa -based Language Model . arXiv:2001.06286 [cs]. ArXiv: 2001.06286

    Pith/arXiv arXiv 2020

  9. [9]

    Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020 b . https://doi.org/10.18653/v1/2020.findings-emnlp.292 R ob BERT : a D utch R o BERT a-based L anguage M odel . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255--3265, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.292 2020

  10. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies , Volume 1 ( Lon...

    doi:10.18653/v1/n19-1423 2019

  11. [11]

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. https://arxiv.org/abs/2002.06305 Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping . Preprint, arXiv:2002.06305

    Pith/arXiv arXiv 2020

  12. [12]

    Henry He, Johann Frei, and Raphael Scheible-Schmitt. 2025. https://doi.org/10.21203/rs.3.rs-7332811/v1 The word and the way: Strategies for domain-specific BERT pre-training in german medical NLP . ISSN : 2693-5015

    work page doi:10.21203/rs.3.rs-7332811/v1 2025

  13. [13]

    Tim Jansen, Yangling Tong, Victoria Zevallos, and Pedro Ortiz Suarez. 2022. https://arxiv.org/abs/2212.10440 Perplexed by quality: A perplexity-based method for adult and harmful content detection in multilingual heterogeneous web data . Preprint, arXiv:2212.10440

    Pith/arXiv arXiv 2022

  14. [14]

    Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. 2021. https://arxiv.org/abs/2008.02496 Convbert: Improving bert with span-based dynamic convolution . Preprint, arXiv:2008.02496

    Pith/arXiv arXiv 2021

  15. [15]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. https://arxiv.org/abs/1909.10351 Tinybert: Distilling bert for natural language understanding . Preprint, arXiv:1909.10351

    Pith/arXiv arXiv 2020

  16. [16]

    Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. 2023. https://doi.org/10.48550/arXiv.2304.01433 TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddin...

    Pith/arXiv arXiv doi:10.48550/arxiv.2304.01433 2023

  17. [17]

    Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. https://www.aclweb.org/anthology/P07-2045 Moses: Open Source Toolkit for Statistical Machine Translation . In Proceedings of the...

    2007

  18. [18]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 RoBERTa : A Robustly Optimized BERT Pretraining Approach . arXiv:1907.11692 [cs]. ArXiv: 1907.11692

    Pith/arXiv arXiv 2019

  19. [19]

    Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. https://arxiv.org/abs/2509.06888 mmbert: A modern multilingual encoder with annealed language learning . Preprint, arXiv:2509.06888

    Pith/arXiv arXiv 2025

  20. [20]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, \' E ric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. https://www.aclweb.org/anthology/2020.acl-main.645 CamemBERT : a Tasty French Language Model . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7203--7219,...

    2020

  21. [21]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. http://arxiv.org/abs/1904.01038 fairseq: A Fast , Extensible Toolkit for Sequence Modeling . arXiv:1904.01038 [cs]. ArXiv: 1904.01038

    Pith/arXiv arXiv 2019

  22. [22]

    Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. https://doi.org/10.18653/v1/P17-1178 Cross-lingual name tagging and linking for 282 languages . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946--1958, Vancouver, Canada. Association for Com...

    doi:10.18653/v1/p17-1178 2017

  23. [23]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9

    2019

  24. [24]

    Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. https://doi.org/10.18653/v1/P19-1015 Massively multilingual transfer for NER . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151--164, Florence, Italy. Association for Computational Linguistics

    doi:10.18653/v1/p19-1015 2019

  25. [25]

    Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, and Martin Boeker. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1183 G ott BERT : a pure G erman language model . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21237--21250, Miami, ...

    doi:10.18653/v1/2024.emnlp-main.1183 2024

  26. [26]

    Raphael Scheible-Schmitt and Johann Frei. 2025. https://arxiv.org/abs/2506.11903 Geistbert: Breathing life into german nlp . Preprint, arXiv:2506.11903

    Pith/arXiv arXiv 2025

  27. [27]

    Stefan Schweter. 2020. https://doi.org/10.5281/zenodo.3770924 Berturk - bert models for turkish . https://doi.org/10.5281/zenodo.3770924. Version 1.0.0, Zenodo

    doi:10.5281/zenodo.3770924 2020

  28. [28]

    Stefan Schweter. 2025. https://doi.org/10.5281/zenodo.14963493 BERTurk v2 . https://doi.org/10.5281/zenodo.14963493. Version 2.0.0, Zenodo

    work page doi:10.5281/zenodo.14963493 2025

  29. [29]

    Nuri Tas. 2024. https://arxiv.org/abs/2401.03515 Roberturk: Adjusting roberta for turkish . Preprint, arXiv:2401.03515

    Pith/arXiv arXiv 2024

  30. [30]

    Cagri Toraman, Eyup Halit Yilmaz, Furkan S ahinu c, and Oguzhan Ozcelik. 2023. https://doi.org/10.1145/3578707 Impact of tokenization on language models: An analysis for turkish . ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1–21

    doi:10.1145/3578707 2023

  31. [31]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. https://doi.org/10.1162/tacl_a_00321 BL i MP : The benchmark of linguistic minimal pairs for E nglish . Transactions of the Association for Computational Linguistics, 8:377--392

    doi:10.1162/tacl_a_00321 2020

  32. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  33. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.