Pith. sign in

REVIEW 2 major objections 6 minor 75 references

skLEP: A Slovak General Language Understanding Benchmark

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims skLEP is the first GLUE-style benchmark for Slovak natural language understanding, with nine tasks at token, sentence-pair, and document levels, and a first fine-tuned comparison of fourteen pretrained models.

desk verdict A solid, honest first Slovak GLUE-style benchmark, with one load-bearing translation-quality question that needs more evidence before the rankings are taken as settled. read the letter →

arxiv 2506.21508 v1 pith:EOF53YD5 submitted 2025-06-26 cs.CL cs.AIcs.IRcs.LG

Marek Šuppa , Andrej Ridzik , Daniel Hládek , Tomáš Javůrek , Viktória Ondrejová , Kristína Sásiková , Martin Tamajka , Marián Šimko This is my paper
classification cs.CLcs.AIcs.IRcs.LG
keywords SlovaknaturallanguageunderstandingGLUE-stylebenchmarkpretrainedmodelevaluationmachinetranslationpipelineRelativeErrorReductiontoken-levelandsentence-pairtasksmid-resourceNLPleaderboard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Slovak, a mid-resource language with about ten million speakers, has had no standardized suite for measuring natural language understanding, which makes it hard to compare the several Slovak-specific language models that have recently appeared. The paper claims to fill that gap with skLEP, a GLUE-style benchmark of nine tasks covering token-level, sentence-pair, and document-level understanding, built partly from original Slovak datasets and partly from English tasks translated with native-speaker post-editing. It then reports the first systematic fine-tuned comparison of fourteen Slovak, multilingual, and English pretrained models on these tasks, using relative error reduction against SlovakBERT as the headline metric. The result a sympathetic reader would take away: the multilingual mDeBERTaV3Base is the current leader, but the older SlovakBERT still beats most newer alternatives, and the hardest open targets are question answering and textual entailment.

What carries the argument

The central object is the benchmark itself: nine Slovak NLU tasks with fixed splits and task-specific metrics (Macro F1, accuracy, Pearson correlation), assembled by mixing curated original datasets with machine-translated English tasks. Two mechanisms carry the argument. The first is the translation pipeline, which uses DeepL for most datasets and the open-weights MADLAD-400-3B for the much larger NLI corpus, validated by a five-system translation quality study and by native-speaker post-editing of the development and test sets. The second is the aggregation metric, Relative Error Reduction (RER), taken from the DUMB benchmark, which normalizes each task score by the baseline error rate so that gains on near-solved tasks like UD count more than equal gains on harder tasks like QA; this is what lets the paper rank models on balanced performance rather than on raw averages.

What would settle it

Re-run the paper's Appendix B relabeling experiment at scale: have native Slovak annotators independently re-label a few thousand random instances from the RTE, NLI, and STS training and test sets, then compare model rankings computed on the corrected labels against the published ones. If the label-change rate stays in the single digits and the rankings are unchanged, the translation premise holds; if corrected labels move which model leads or meaningfully reorder the table, the benchmark's conclusions on the translated tasks would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that skLEP is the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding. It comprises nine tasks: part-of-speech tagging, two named-entity-recognition datasets (Universal NER's Slovak subset and the hand-annotated WikiGoldSK), textual entailment, natural language inference, semantic textual similarity, hate speech classification, sentiment analysis, and the SK-QuAD question answering dataset. Three of the nine (RTE, NLI, STS) were produced by machine-translating established English resources and post-editing the development and test splits with native speakers, while the rest are curated original Slovak datasets. On top of the benchmark, the paper presents the first systematic evaluation of fourteen pretrained models fine-tuned on all tasks, reporting both absolute scores and Relative Error Reduction with SlovakBERT as baseline; mDeBERTaV3Base achieves the highest mean score and a 6.43 percentage-point error reduction, and only two models beat SlovakBERT's average. The authors also release the data, a fine-tuning and evaluation toolkit, and a public leaderboard.

Load-bearing premise

The load-bearing premise is that machine-translated English sentences with their original labels measure Slovak natural language understanding rather than translationese; the paper itself concedes that the RTE, NLI, and STS training sets were not manually corrected, and its own 100-sample checks found 2 to 5 percent of labels changed by translation errors, so if those errors shift difficulty or labels systematically across the full corpora, model rankings on these three tasks could change.

Editorial extensions

If this is right

  • Slovak NLP gains a public, standardized evaluation suite with a leaderboard, so future models can be compared on identical ground instead of ad-hoc task choices.
  • The baselines fix a concrete state of the art: mDeBERTaV3Base leads with 6.43 percentage points of RER over SlovakBERT, and any new Slovak model can be measured against this.
  • SlovakBERT's strong showing suggests language-specific pretraining still pays off for mid-resource languages, even against much newer multilingual and English models.
  • Question answering (F1 mostly below 75) and textual entailment (accuracy mostly below 70) are the clearest open problems the benchmark exposes.
  • The reported translation-error rates (2 to 5 percent label changes, STS mean absolute error 0.69) give users a quantitative bound on the noise in the three translated tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's pipeline, translate, post-edit, spot-check label stability, is a reusable recipe for bootstrapping benchmarks in other mid-resource languages, and reporting translation-error rates the way this paper does could become a norm for translated benchmarks.
  • Because the NLI training set was translated with the weaker open-weights model while its development and test sets were post-edited, models fine-tuned on it may internalize translationese patterns; a natively collected Slovak NLI corpus could shift rankings on that task, a testable prediction the paper does not make.
  • The RER aggregation embeds a value judgment, rewarding balanced competence over peak performance on easy tasks; if other benchmarks adopt it, it changes which models win, so the choice deserves explicit discussion wherever benchmarks are compared.
  • The leaderboard's policy of fully public test sets trades away the protection that keeps comparable English benchmarks uncontaminated; as skLEP gains users, leakage of test items into pretraining data may become a concern.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces skLEP, a nine-task benchmark for Slovak natural language understanding, covering token-level (UD, UNER, WGSK), sentence-pair (RTE, NLI, STS), and document-level (HS, SA, QA) tasks. Six tasks are curated from existing Slovak resources or newly created datasets; three tasks (RTE, NLI, STS) are machine-translated from English with native-speaker post-editing of the validation and test sets. The authors evaluate 14 pretrained encoder-only models under a uniform fine-tuning protocol with an extensive hyperparameter search (4,024 runs), report both absolute scores and relative error reduction (RER) against SlovakBERT, and release a toolkit and leaderboard. The paper claims to provide the first comprehensive GLUE-style benchmark for Slovak NLU.

Significance. If the translation-quality concerns are resolved, skLEP would fill a clear gap for Slovak NLP: it provides a standardized evaluation suite, the first systematic comparison of Slovak-specific and multilingual encoder models on Slovak, and a public leaderboard that can drive future research. The work has notable strengths: duplicate removal, explicit native-speaker post-editing for test sets, transparent documentation of the hyperparameter search, and a published translation-quality assessment. The open-source toolkit and detailed appendices support reproducibility. The benchmark's novelty as the first Slovak GLUE-style suite appears justified by the related-work survey.

major comments (2)
  1. [Section 3.2, Section 4, Appendix B] The validity of the three machine-translated tasks (RTE, NLI, STS) is the load-bearing premise for the claim that skLEP measures Slovak NLU, but the supporting evidence is insufficient. The relabeling check in Appendix B uses only 100 samples per task, with co-author annotators, and the NLI training set was translated by MADLAD-400-3B, the worst-ranked system in the paper's own quality evaluation, without post-editing. The Limitations section admits that these training sets 'have not been manually corrected.' With 5% label-changing errors in NLI, the 392,702 training instances could contain tens of thousands of mislabeled pairs, which may shift fine-tuning outcomes and model rankings. The paper should provide a larger, independent error study (e.g., several hundred samples per task with non-author annotators), report label-error rates on the test sets separately, and analyze the sensitivity of model rankings to the inclusion of the translated tasks (e.g., by reranking models on the six non-translated tasks alone). Without such analysis, the claim that all nine tasks measure Slovak NLU is not fully supported.
  2. [Section 4, Tables 3 and 4] The decision to use MADLAD-400-3B for the NLI training corpus is justified solely by cost, despite this system ranking lowest in the paper's own translation-quality assessment (mean rank 2.54 vs. DeepL's 1.81; fluency 3.48 vs. 3.70). The paper should demonstrate that this choice does not materially affect the NLI task's validity, for example by comparing models fine-tuned on a sample of MADLAD-translated data against the same models fine-tuned on DeepL-translated or post-edited data, or by quantifying the downstream effect of the observed label noise. As it stands, NLI scores may reflect robustness to machine-translation artifacts rather than Slovak inference ability.
minor comments (6)
  1. [Abstract and Section 6] The abstract states that the benchmark data, toolkit, and leaderboard are released, but Section 6 says 'We will release all software and data upon acceptance' and that translated datasets are 'not yet public.' These statements should be aligned; providing anonymized access to the data during review would strengthen the reproducibility claim.
  2. [Table 5] The header row of Table 5 is garbled (e.g., 'A VG', 'RERAvg', 'RERF1' run together), making the column structure difficult to parse. The table should be reformatted so each task and metric is clearly separated.
  3. [Appendix A] Appendix A mentions 'MobileBERT models on the QA task' as unsupported, but no MobileBERT model appears in the paper's model list or in Table 5. This appears to be a leftover from an earlier draft and should be removed or corrected.
  4. [Appendix C] The post-editing quality experiment uses 30 post-edited and 30 non-post-edited samples, but the post-edited set was 'manually chosen' for inclusion. The statement that 'only 15/60 samples needed post-editing' is therefore not an estimate of the population rate and should be phrased accordingly to avoid overinterpretation.
  5. [Section 5.4] The RER values are computed relative to SlovakBERT, a model developed by an overlapping group. The paper should note explicitly that RER is baseline-dependent and that negative RER does not imply below-chance performance; a brief sensitivity check with an alternative baseline would strengthen the interpretation.
  6. [Section 3.1] The WGSK description says the dataset addresses limitations of 'silver-standard resources' but does not specify what those limitations are or how WikiGoldSK's gold-standard quality was verified. A sentence clarifying the annotation procedure and quality checks would improve transparency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: skLEP scores are empirical measurements on public or disclosed datasets, and the only definitional identity is the explicit RER baseline normalization for SlovakBERT.

full rationale

The paper's central outputs are measured task scores from fine-tuning 14 pretrained models on nine datasets; no parameter is fitted and then reported as a prediction. The three machine-translated tasks (RTE, NLI, STS) inherit labels from the original English benchmarks and are evaluated on the resulting Slovak text; translationese is a validity threat that the authors disclose in the Limitations, but it is not a circular step because the scores are empirically measured rather than derived from the translation-quality check. The RER metric sets SlovakBERT's RER to 0.0 by definition ('We use SlovakBERT as our baseline'), yet this is an explicit normalization and the paper also reports mean absolute scores, so no conclusion is forced by construction. Self-citations to UNER, WGSK, SK-QuAD, SA, and SlovakBERT point to public, externally evaluable resources; while the overlap with the author group warrants scrutiny, these resources are not the conclusions of the paper and the benchmark measurements are independent of them. No equation, fitted parameter, or uniqueness claim in the paper reduces to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The benchmark's validity rests on the adequacy of machine translation with retained labels, the fairness of the fine-tuning comparison, and the cost-driven choice to use a weaker translator for the largest corpus. No physical entities are postulated; the only new artifact is the skLEP resource itself, whose public release is promised but not yet available for verification.

free parameters (3)
  • Per-task hyperparameters (learning rate, epochs, warmup) selected by grid search on dev splits = e.g., LR 1e-5 to 1e-4, epochs 1-10 depending on task and model
    The reported test scores depend on choosing the best config per model-task on the dev set; other configs could shift rankings.
  • RER baseline model (SlovakBERT) = SlovakBERT scores
    Relative Error Reduction is computed against SlovakBERT, a model built by several of the same authors; selecting a different baseline would change all RER values.
  • Dev split sizes for HS and QA = dev set matched to test set size (1,319 and 9,583)
    The validation splits were created by sampling from the original training sets to match test size; this choice influences hyperparameter selection.
assumptions (4)
  • domain assumption Machine-translated datasets retain the validity of original English labels in Slovak
    Used in Section 3.2 for RTE, NLI, STS; partially tested in Appendix B where 2-5% of relabeled samples changed due to translation errors, and the NLI training set was never manually corrected.
  • domain assumption Fine-tuning with the searched hyperparameters gives a fair, comparable measure of model capability on Slovak
    Stated in Section 5 and Limitations: the NLI search was restricted by compute budget, and better configs may exist; results without variance estimates are treated as stable.
  • ad hoc to paper MADLAD-400-3B translation quality is adequate for the large NLI training corpus even though it scored worst in the ranking experiment
    Section 4: DeepL was chosen for translation but MADLAD-400-3B was used for NLI to control cost; the paper estimates 5% training-set label errors from translation.
  • domain assumption Public test sets will not be contaminated by future model training
    Section 6 acknowledges the risk of test-set leakage because the split is public.
invented entities (1)
  • skLEP benchmark suite and leaderboard
    purpose: Provide standardized Slovak NLU evaluation
    The manuscript includes samples and promised URLs, but states datasets and leaderboard are not yet public; independent verification currently rests on the paper's own examples and tables.

how reviews work

0 comments
Cite this review

Pith. "Pith review of skLEP: A Slovak General Language Understanding Benchmark." pith.science (2026). https://pith.science/paper/EOF53YD5

@misc{pith2026250621508,
  author       = {Pith},
  title        = {Pith review of: skLEP: A Slovak General Language Understanding Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOF53YD5}},
  note         = {Machine review of arXiv:2506.21508}
}
read the original abstract

In this work, we introduce skLEP, the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding (NLU) models. We have compiled skLEP to encompass nine diverse tasks that span token-level, sentence-pair, and document-level challenges, thereby offering a thorough assessment of model capabilities. To create this benchmark, we curated new, original datasets tailored for Slovak and meticulously translated established English NLU resources. Within this paper, we also present the first systematic and extensive evaluation of a wide array of Slovak-specific, multilingual, and English pre-trained language models using the skLEP tasks. Finally, we also release the complete benchmark data, an open-source toolkit facilitating both fine-tuning and evaluation of models, and a public leaderboard at https://github.com/slovak-nlp/sklep in the hopes of fostering reproducibility and drive future research in Slovak NLU.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 33 canonical work pages

  1. [1]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856

    arXiv 2019

  2. [2]

    Lukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Arkadiusz Janz, Piotr Szyma \'n ski, Marcin W a troba, Miko aj Morzy, et al. 2022. This is the way: designing and compiling lepiszcze, a comprehensive nlp benchmark for polish. Advances in Neural Information Processing Systems, 35:21805--21818

    work page 2022

  3. [3]

    Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 1. Citeseer

    work page 2006

  4. [4]

    Valerio Basile, Livio Bioglio, Alessio Bosca, Cristina Bosco, and Viviana Patti. 2023. Uinauil: A unified benchmark for italian natural language understanding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 348--356

    work page 2023

  5. [5]

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1

    work page 2009

  6. [6]

    Aleksandrs Berdi c evskis, Gerlof Bouma, Robin Kurtz, Felix Morger, Joey \"O hman, Yvonne Adesam, Lars Borin, Dana Dann \'e lls, Markus Forsberg, Tim Isbister, et al. 2023. Superlim: A swedish language understanding evaluation benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8137--8153

    2023

  7. [7]

    Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explain the cross-lingual capabilities of english pretrained models. arXiv preprint arXiv:2204.08110

    arXiv 2022

  8. [8]

    Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?

    Vicent Briva-Iglesias, Joao Lucas Cavalheiro Camargo, and Gokhan Dogru. 2024. Large language models" ad referendum": How good are they at machine translation in the legal domain? arXiv preprint arXiv:2402.07681

    work page Pith review arXiv 2024

Show all 75 references
  1. [9]

    Vicent Briva-Iglesias, Sharon O’Brien, and Benjamin R Cowan. 2023. The impact of traditional and interactive post-editing on machine translation user experience, quality, and productivity. Translation, Cognition & Behavior, 6(1):60--86

    2023

  2. [10]

    Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055

    2017 arXiv

  3. [11]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  4. [12]

    Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053

    2018 arXiv

  5. [13]

    Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672

    2022 arXiv

  6. [14]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177--190. Springer

    2005

  7. [15]

    Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Ba \ n \'o n, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ram \' rez-S \'a nchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and J \"o rg Tiedemann. 2024. https://aclanthology.org/2024.lr...

    2024

  8. [16]

    Wietse de Vries, Martijn Wieling, and Malvina Nissim. 2023. Dumb: A benchmark for smart evaluation of dutch models. arXiv preprint arXiv:2305.13026

    2023 arXiv

  9. [17]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  10. [18]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  11. [19]

    Dávid Držík and František Forgac. 2024. https://doi.org/10.7717/peerj-cs.2465 Slovak morphological tokenizer using the byte-pair encoding algorithm . PeerJ Computer Science, 10:e2465

    2024 doi

  12. [20]

    Stefan Daniel Dumitrescu, Petru Rebeja, Beata Lorincz, Mihaela Gaman, Andrei Avram, Mihai Ilie, Andrei Pruteanu, Adriana Stan, Lorena Rosia, Cristina Iacobescu, et al. 2021. Liro: Benchmark and leaderboard for romanian language tasks. In Thirty-fifth Conference on Neural Infor...

    2021

  13. [21]

    AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2022. Orca: A challenging benchmark for arabic language understanding. arXiv preprint arXiv:2212.10758

    2022 arXiv

  14. [22]

    Kawin Ethayarajh and Dan Jurafsky. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.393 Utility is in the eye of the user: A critique of NLP leaderboards . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4846--4853, Onl...

    2020 doi

  15. [23]

    Katar \' na Gajdo s ov \'a , M \'a ria S imkov \'a , and et al. 2016. http://hdl.handle.net/11234/1-1822 Slovak dependency treebank . LINDAT / CLARIAH - CZ digital library at the Institute of Formal and Applied Linguistics ( \'U FAL ), Faculty of Mathematics and Physics, Charl...

    2016

  16. [24]

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1--9

    2007

  17. [25]

    Daniil Gurgurov, Rishu Kumar, and Simon Ostermann. 2025. https://arxiv.org/abs/2409.18193 Gremlin: A repository of green baseline embeddings for 87 low-resource languages injected with multilingual graph knowledge . Preprint, arXiv:2409.18193

    2025 arXiv

  18. [26]

    Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Veselin Stoyanov, Ivan Koychev, Preslav Nakov, and Dragomir Radev. 2023. https://doi.org/10.18653/v1/2023.acl-long.487 bg GLUE : A B ulgarian general language understanding evaluation...

    2023 doi

  19. [27]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . Preprint, arXiv:2111.09543

    2021 arXiv

  20. [28]

    Daniel Hládek, Ján Staš, Jozef Juhár, and Tomáš Koctúr. 2023. https://doi.org/10.1109/ACCESS.2023.3262308 Slovak dataset for multilingual question answering . IEEE Access, 11:32869--32881

    2023

  21. [29]

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411--4421. PMLR

    2020

  22. [30]

    Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. https://doi.org/10.18653/v1/2020.acl-main.560 The state and fate of linguistic diversity and inclusion in the NLP world . In Proceedings of the 58th Annual Meeting of the Association for Co...

    2020 doi

  23. [31]

    Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, et al. 2021. Parsinlu: a suite of language understanding challenges for persian. Transactions of the Association ...

    2021

  24. [32]

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adi...

    2021 doi

  25. [33]

    Moshe Koppel and Noam Ordan. 2011. https://aclanthology.org/P11-1132/ Translationese and its dialects . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1318--1326, Portland, Oregon, USA. Association...

    2011

  26. [34]

    Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2024. Madlad-400: A multilingual and document-level large audited dataset. Advances in Neural Information Processing Systems, 36

    2024

  27. [35]

    Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. Jglue: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957--2966

    2022

  28. [36]

    Hang Le, Lo \" c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabb \'e , Laurent Besacier, and Didier Schwab. 2019. Flaubert: Unsupervised language model pre-training for french. arXiv preprint arXiv:1912.05372

    2019 arXiv

  29. [37]

    Jan Lehe c ka and Jan S vec. 2021. https://doi.org/10.1007/978-3-030-89579-2_3 Comparison of czech transformers on text classification tasks . In Statistical Language and Speech Processing, pages 27--37, Cham. Springer International Publishing

    2021 doi

  30. [38]

    Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario S a s ko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh,...

    2021

  31. [39]

    Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.813 XLM - V : Overcoming the vocabulary bottleneck in multilingual masked language models . In Proceedings of...

    2023 doi

  32. [40]

    Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint arXiv:2004.01401

    2020 arXiv

  33. [41]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv e-prints, pages arXiv--1907

    2019

  34. [42]

    Ilya Loshchilov and Frank Hutter. 2017. https://arxiv.org/abs/1711.05101 Fixing weight decay regularization in adam . CoRR, abs/1711.05101

    2017 arXiv

  35. [43]

    Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021. Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking. Advances in Neural Information Processing Systems, ...

    2021

  36. [44]

    Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, B \"o rje Karlsson, Peiqin Lin, Nikola Ljube s i \'c , Lester James Miranda, Barbara Plank, Arij Riabi, and Yuval Pinter. 2024. https://doi.org/10.18653/v1/2024.naacl-long.243 Universa...

    2024 doi

  37. [45]

    Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276--282

    2012

  38. [46]

    Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman

    Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Haji c , Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. https://aclanthology.org/2020.lrec-1.497/ U niversal D ependencies v2: An evergrowing multilingual treebank c...

    2020

  39. [47]

    Tom \'a s Os \'o rio, Bernardo Leite, Henrique Lopes Cardoso, Lu \' s Gomes, Jo \ a o Rodrigues, Rodrigo Santos, and Ant \'o nio Branco. 2024. Portulan extraglue datasets and models: Kick-starting a benchmark for the neural processing of portuguese. arXiv preprint arXiv:2404.05333

    2024 arXiv

  40. [48]

    Sungjoon Park. 2021. Klue: Korean language understanding evaluation. arXiv preprint arXiv:2105.09680

    2021 arXiv

  41. [49]

    Samuel Pecar, Marian Simko, and Maria Bielikova. 2019. https://doi.org/10.18653/v1/W19-3716 Improving sentiment classification in S lovak language . In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 114--119, Florence, Italy. Association for...

    2019 doi

  42. [50]

    Jan Pfister and Andreas Hotho. 2024. Supergleber: German language understanding evaluation benchmark. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), page...

    2024

  43. [51]

    Mat \'u s Pikuliak, S tefan Grivalsk \`y , Martin Kon \^o pka, Miroslav Bl s t \'a k, Martin Tamajka, Viktor Bachrat \`y , Mari \'a n S imko, Pavol Bal \'a z ik, Michal Trnka, and Filip Uhl \'a rik. 2021. Slovakbert: Slovak masked language model. arXiv preprint arXiv:2109.15254

    2021 arXiv

  44. [52]

    Jakub Piskorski, Lidia Pivovarova, Jan S najder, Josef Steinberger, and Roman Yangarber. 2017. The first cross-lingual challenge on recognition, normalization and matching of named entities in slavic languages. In Workshop on Balto-Slavic Natural Language Processing, pages 76-...

    2017

  45. [53]

    Giovanni Puccetti, Maria Cassese, and Andrea Esuli. 2025. https://aclanthology.org/2025.coling-main.453/ The invalsi benchmarks: measuring the linguistic and mathematical understanding of large language models in I talian . In Proceedings of the 31st International Conference o...

    2025

  46. [54]

    Carlos Rodriguez-Penagos, Carme Armentano-Oller, Marta Villegas, Maite Melero, Aitor Gonzalez, Ona de Gibert Bonet, and Casimiro Carrino Pio. 2021. The catalan language club. arXiv preprint arXiv:2112.01894

    2021 arXiv

  47. [55]

    Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, et al. 2021. Xtreme-r: Towards more challenging and nuanced multilingual evaluation. arXiv preprint arXiv:2104.07412

    2021 arXiv

  48. [56]

    Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. 2020. Klej: Comprehensive benchmark for polish language understanding. arXiv preprint arXiv:2005.00630

    2020 arXiv

  49. [57]

    David Samuel, Andrey Kutuzov, Lilja vrelid, and Erik Velldal. 2023 a . https://doi.org/10.18653/v1/2023.findings-eacl.146 Trained on 100 million words and still in shape: BERT meets B ritish N ational C orpus . In Findings of the Association for Computational Linguistics: EACL...

    2023 doi

  50. [58]

    David Samuel, Andrey Kutuzov, Samia Touileb, Erik Velldal, Lilja vrelid, Egil R nningstad, Elina Sigdel, and Anna Palatkina. 2023 b . Norbench--a benchmark for norwegian language models. arXiv preprint arXiv:2305.03880

    2023 arXiv

  51. [59]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108

    2019 arXiv

  52. [60]

    Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. Russiansuperglue: A russian language understanding evaluation benchmark. arXiv preprint a...

    2020 arXiv

  53. [61]

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2020. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13693--13696

    2020

  54. [62]

    David Suba, Marek Suppa, Jozef Kubik, Endre Hamerlik, and Martin Takac. 2023. https://doi.org/10.18653/v1/2023.bsnlp-1.16 W iki G old SK : Annotated dataset, baselines and few-shot learning experiments for S lovak named entity recognition . In Proceedings of the 9th Workshop o...

    2023 doi

  55. [63]

    Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, and Douwe Kiela. 2022. https://doi.org/10.18653/v1/2022.acl-demo.17 Dynatask: A framework for creating dynamic AI benchmark tasks . In ...

    2022 doi

  56. [64]

    Antonio Toral. 2019. https://aclanthology.org/W19-6627/ Post-editese: an exacerbated translationese . In Proceedings of Machine Translation Summit XVII: Research Track, pages 273--281, Dublin, Ireland. European Association for Machine Translation

    2019

  57. [65]

    Gorka Urbizu, I \ n aki San Vicente, Xabier Saralegi, Rodrigo Agerri, and Aitor Soroa. 2022. Basqueglue: A natural language understanding benchmark for basque. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1603--1612

    2022

  58. [66]

    Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  59. [67]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32

    2019

  60. [68]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. https://arxiv.org/abs/2412.13663 Smarter, better, f...

    2024 arXiv

  61. [69]

    Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, et al. 2020. Indonlu: Benchmark and resources for evaluating indonesian natural language understanding. arXiv preprint...

    2020 arXiv

  62. [70]

    Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426

    2017 arXiv

  63. [71]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  64. [72]

    Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaowei...

    2020 doi

  65. [73]

    Ale s Z agar and Marko Robnik- S ikonja. 2022. Slovene superglue benchmark: translation and evaluation. arXiv preprint arXiv:2202.04994

    2022 arXiv

  66. [74]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  67. [75]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.