Pith. sign in

REVIEW 3 major objections 4 minor 45 references

The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The AI Language Proficiency Monitor tracks LLM performance across up to 200 languages with an auto-updating leaderboard.

desk verdict The system is real and open, but 10-example sampling and unvalidated machine translation put the headline proficiency claim on shaky ground. read the letter →

arxiv 2507.08538 v1 pith:YI3ZOHVZ submitted 2025-07-11 cs.CL

David Pomerenke , Jonas Nothnagel , Simon Ostermann This is my paper
classification cs.CL
keywords multilingualbenchmarklow-resourcelanguagesLLMevaluationleaderboardlanguageproficiencyscoremachinetranslationFLORES+NLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The AI Language Proficiency Monitor is an attempt to turn scattered multilingual evaluation datasets into one continuously updated measure of how well large language models handle the world's languages, with a deliberate emphasis on low-resource languages. The paper claims that a single open pipeline can track models across up to 200 languages by combining translation, classification, question answering, math, and factuality tasks and compressing them into an aggregate Language Proficiency Score. If the claim holds, researchers, developers, and policymakers gain a practical, open tool for seeing which languages are underserved by current AI and which models help close the gap. The system is designed to auto-update daily and to present results through an accessible dashboard with maps, trends, and per-language leaderboards.

What carries the argument

The central object is the Language Proficiency Score, defined as the mean of min-max-normalized task metrics across translation, classification, question answering, math, and factuality. It is carried by an auto-updating evaluation pipeline that runs daily, pulls models through commercial and open inference APIs, and evaluates each model-task-language combination in a few-shot, multiple-choice format with language-agnostic prompts. To keep scores comparable across languages, the pipeline restricts all tasks to parallel rows where datasets overlap, prefers human-translated items, and machine-translates missing languages with a commercial neural translation service, scoring translation with SpBLEU, a SentencePiece-based metric trained equally on FLORES+ languages.

What would settle it

Re-score a set of low-resource languages using professionally human-translated versions of the same benchmark items and compare per-language Language Proficiency Scores with the current machine-translated results; if the ranking of models changes materially for those languages, the monitor is partly measuring translation artifacts rather than model proficiency.

Watch

Extended reading notes

Core claim

The paper's central claim is that a unified, auto-updating benchmark can systematically assess LLM proficiency across up to 200 languages, going beyond regional or one-off evaluations. The monitor combines FLORES+ and SIB-200 for translation and classification; MMLU, ARC, and TruthfulQA for question answering and factuality; and GSM8K/MGSM for mathematics, preferring human-translated versions and filling gaps with automatic machine translation. Every task score is min-max normalized and averaged into a Language Proficiency Score, reported per model and per language. The paper also presents descriptive analyses, including a country-level map, temporal trends, and comparisons of high- versus low-resource languages, arguing these help identify capacity gaps.

Load-bearing premise

The load-bearing premise is that machine-translated benchmark items preserve the difficulty and construct of the English originals in each target language, and that the ten sampled examples per model-task-language combination are enough for stable scores.

Editorial extensions

If this is right

  • The daily pipeline means the leaderboard always reflects the latest models, rather than a single static evaluation.
  • Per-language and per-task scores will let developers spot concrete gaps, such as strong translation but weak mathematical reasoning in a given language.
  • The country-level proficiency map gives policymakers a direct view of where AI capacity is most uneven.
  • Because the same parallel items are used across languages, reported cross-language differences are meant to be comparable rather than metric artifacts.
  • Local developers can submit their own open-weights models through public inference endpoints, creating a route for community-built models to be benchmarked.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate, untested extension would be to report per-language translation-quality diagnostics, since scores for machine-translated languages may partly reflect translation fidelity rather than model ability.
  • The ten-example sampling rate implies that score differences between closely ranked models should be accompanied by confidence intervals; without them, small gaps in the leaderboard may not be stable.
  • A further extension would replace translated English items with tasks originally written in each language and grounded in local knowledge, testing whether the monitor's scores predict real-world usefulness in low-resource contexts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents the AI Language Proficiency Monitor, an open-source, auto-updating multilingual benchmark and dashboard that aggregates existing datasets (FLORES+, SIB-200, MMLU variants, ARC, GSM8K, TruthfulQA) to evaluate LLMs across up to 200 languages. The authors define a Language Proficiency Score as the mean of min-max normalized task metrics, run daily evaluations via OpenRouter and HuggingFace APIs, and provide leaderboards, maps, and trend visualizations. They also report qualitative stakeholder feedback and state current limitations, including 10-instance sampling and machine-translated data.

Significance. If the measurement pipeline were validated, the system would fill a practical gap by providing continuously updated multilingual model rankings with broader language coverage than prior leaderboards, and its open-source architecture plus community submission mechanism are concrete strengths. However, the paper's headline claims about tracking proficiency across many low-resource languages rest on statistical and translation-quality assumptions that are not currently established.

major comments (3)
  1. [Section 5.1 and Section 3.4] The automated evaluation samples only 10 instances per model-task-language, and the Language Proficiency Score is a mean of per-task accuracies. For an accuracy task, each instance contributes 10 percentage points, so the task-level estimates have enormous standard errors; yet Figures 2-3 and the leaderboard report these scores without confidence intervals or significance testing. The reported rankings by language and by model are therefore not distinguishable from sampling noise. Please either scale up the evaluation, report uncertainty intervals on the LPS, or explicitly restrict the claims to the specific 10-item sample.
  2. [Section 3.1 and Table 1] Large fractions of the evaluation items in MMLU (61 languages), ARC (94), GSM8K (73), and TruthfulQA (94) are produced by Google Cloud Translate, and the paper provides no validation that these translations preserve item difficulty or construct, nor any per-language measure of translation quality. Since machine translation quality is known to vary strongly with language resource level, the language-level scores in Figure 3 are confounded: differences attributed to model proficiency may reflect translation artifacts. The authors should report translation quality checks (e.g., human evaluation on a sample, back-translation scores, or agreement with existing human-translated subsets) and either filter low-quality translations or discuss the impact on the rankings.
  3. [Section 3.2] All prompts are kept in English with minimal language-agnostic instructions, so the task is presented in a language different from the target language for most languages. This design choice can systematically disadvantage languages for which the model has weaker instruction-following ability, and the paper offers no experiments comparing English-prompt with native-prompt performance. Since the goal is to measure 'understanding of a given language,' the authors should either justify this choice empirically or acknowledge that the metric reflects cross-lingual transfer from English prompts rather than language-specific proficiency.
minor comments (4)
  1. [Section 1] The sentence 'A number of multilingual LLM benchmark exists' should be 'A number of multilingual LLM benchmarks exist', and the Table 1 caption contains 'The table cover' which should be 'The table covers'.
  2. [Section 3.1] The word 'overproporionate' is a misspelling of 'disproportionate'.
  3. [Table 1] The meanings of '/usr' and '/cg' in the Translation column are not defined until the caption text; consider placing explicit 'human' and 'machine' labels directly in the table header or adding a footnote.
  4. [Section 5.2] The future directions paragraph is brief; consider specifying concrete plans for scaling up the 10-instance sampling and for validating translation quality, since these are the central limitations identified in Section 5.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Language Proficiency Score is a transparent aggregation of externally sourced benchmark tasks, with no fitted parameter, self-citation chain, or definitional loop.

full rationale

The paper's central claim is the construction of a multilingual leaderboard from existing external datasets (FLORES+, MMLU, GSM8K, TruthfulQA, ARC). The Language Proficiency Score is defined as "the mean of the individual task metrics" after min-max normalization, and every task metric is computed directly from model outputs on those datasets; no parameter is fitted to the resulting scores, no task is selected or weighted by the score it produces, and the ranking is not presupposed by the definition. The machine-translation step (Section 3.1) and the 10-instance sampling limit (Section 5.1) are threats to measurement validity and statistical stability, but they are not circular: the pipeline would produce the same scores under better translation or larger samples, and the score does not reduce to the translation or sampling procedure. Citations to prior work are used to source datasets and models, and the only author-overlapping citation (Gurgurov et al., 2025, which includes author Ostermann) appears in related work on embedding baselines and is not load-bearing. There is no self-citation chain, imported uniqueness theorem, or renamed known result; the benchmark is self-contained as an evaluation harness even if its conclusions are limited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central score rests on design choices and external tooling rather than a theoretical derivation: the equal task weighting, the 10-sample evaluation budget, the use of machine-translated items, and the trust in SpBLEU and API evaluation all enter the score. None is fitted to data, but each is an unvalidated assumption that shapes the leaderboard.

free parameters (3)
  • Equal task weights in Language Proficiency Score = mean across five normalized task scores
    Defined in Section 3.4. The choice of equal weighting is unvalidated; changing weights would reorder models and languages.
  • Evaluation sample size = 10 instances per model-task-language
    Disclosed in Section 5.1. Ten items per cell produce high-variance scores, especially for multiple-choice tasks and low-resource languages.
  • Min-max normalization reference = per-task minimum and maximum observed on the current leaderboard
    Introduced in Section 3.4. Because normalization is relative to the currently evaluated model set, adding new models can rescale every language's score over time.
assumptions (5)
  • domain assumption Google-translated benchmark items preserve the difficulty and construct of the original English items.
    Used in Section 3.1 to fill languages without human-translated data; translations can introduce artifacts that are uncorrelated with model language ability.
  • ad hoc to paper Ten sampled items per model-task-language are representative of the full benchmark.
    Section 5.1 limits runs to 10 instances; no analysis shows this is enough for stable estimates on high-variance tasks.
  • domain assumption SpBLEU is comparably fair across languages.
    Section 3.2 selects SpBLEU for translation scoring; the authors note BLEU and BertScore biases, and SpBLEU's SentencePiece tokenizer is trained on FLORES+ languages, but residual bias is not quantified.
  • ad hoc to paper Min-max normalized scores can be averaged across heterogeneous tasks into a meaningful aggregate.
    Section 3.4 defines the aggregate score as a mean of normalized task metrics; no evidence links this aggregate to real-world language ability.
  • domain assumption API evaluation through OpenRouter endpoints with the no-training setting prevents data contamination.
    Section 3.3 relies on OpenRouter's no-training functionality to avoid contamination; this is a trust assumption about third-party providers.
invented entities (1)
  • Language Proficiency Score (LPS)
    purpose: Single aggregate metric for comparing models and languages
    Defined in Section 3.4 as the mean of min-max normalized task scores; no external validation against human judgments or downstream task performance, and the equal-weighting choice is not independently motivated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks." pith.science (2026). https://pith.science/paper/YI3ZOHVZ

@misc{pith2026250708538,
  author       = {Pith},
  title        = {Pith review of: The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YI3ZOHVZ}},
  note         = {Machine review of arXiv:2507.08538}
}
read the original abstract

To ensure equitable access to the benefits of large language models (LLMs), it is essential to evaluate their capabilities across the world's languages. We introduce the AI Language Proficiency Monitor, a comprehensive multilingual benchmark that systematically assesses LLM performance across up to 200 languages, with a particular focus on low-resource languages. Our benchmark aggregates diverse tasks including translation, question answering, math, and reasoning, using datasets such as FLORES+, MMLU, GSM8K, TruthfulQA, and ARC. We provide an open-source, auto-updating leaderboard and dashboard that supports researchers, developers, and policymakers in identifying strengths and gaps in model performance. In addition to ranking models, the platform offers descriptive insights such as a global proficiency map and trends over time. By complementing and extending prior multilingual benchmarks, our work aims to foster transparency, inclusivity, and progress in multilingual AI. The system is available at https://huggingface.co/spaces/fair-forward/evals-for-every-language.

Figures

Figures reproduced from arXiv: 2507.08538 by the authors.

Figure 1
Figure 1. The map view of the language proficiency [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Language Proficiency Score by language. 3.5 Web Application Our web application is hosted on Huggingface, where it is linked to existing models and datasets. The web app consists of four parts: • Model leaderboard. This table ranks LLMs by their language proficiency and provides the Language Proficiency Score as well as scores for each task. The table can be filtered by open-weights vs API-only models, by cost per o… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 10 canonical work pages

  1. [1]

    Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Beno \^i t Sagot. 2022. https://aclanthology.org/2022.lrec-1.463/ Towards a cleaner document-oriented multilingual crawled corpus . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4344--4355, Marseille, France. European Language Resources Association

    work page 2022

  2. [2]

    Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee

    David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. 2024. https://doi.org/10.48550/arXiv.2309.07445 SIB-200 : A Simple , Inclusive , and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects . Preprint, arXiv:2309.07445

    arXiv doi:10.48550/arxiv.2309.07445 2024

  3. [3]

    David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 8 others. 2025. https://d...

    arXiv doi:10.48550/arxiv.2406.03368 2025

  4. [4]

    Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.258 MEGA : Multilingual evaluation of generative AI . In Proceedings of the 2023 Conference on Empirical Methods in Natural ...

    doi:10.18653/v1/2023.emnlp-main.258 2023

  5. [5]

    Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2024. https://doi.org/10.18653/v1/2024.naacl-long.143 MEGAVERSE : Benchmarking large language models across languages, modalities, models and tasks . In Proceedings of the 2024 Confere...

    doi:10.18653/v1/2024.naacl-long.143 2024

  6. [6]

    Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4971--4980

    work page 2018

  7. [7]

    Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.100 BUFFET : Benchmarking large language models for few-shot cross-lingual transfer . In Proceedings of the 2024 Conference of the North American Chapter of the Associat...

    doi:10.18653/v1/2024.naacl-long.100 2024

  8. [8]

    Alabi, Jonas Kgomo, Tyna Eloundou, Elizabeth Proehl, Kai Chen, Imaan Khadir, Naome A

    Edward Bayes, Israel Abebe Azime, Jesujoba O. Alabi, Jonas Kgomo, Tyna Eloundou, Elizabeth Proehl, Kai Chen, Imaan Khadir, Naome A. Etori, Shamsuddeen Hassan Muhammad, Choice Mpanza, Igneciah Pocia Thete, Dietrich Klakow, and David Ifeoluwa Adelani. https://doi.org/10.48550/arXiv.2412.00948 Uhura: A Benchmark for Evaluating Scientific Question Answering a...

    arXiv doi:10.48550/arxiv.2412.00948

Show all 45 references
  1. [9]

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  2. [10]

    https://doi.org/10.48550/arXiv.1803.05457 Think you have Solved Question Answering ? Try ARC , the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. https://doi.org/10.48550/arXiv.1803.05457 Think you have Solved Question Answering ? Try ARC , the AI2 Reasoning Challenge . Preprint, arXiv:1803.05457

    doi arXiv

  3. [11]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  4. [12]

    https://doi.org/10.48550/arXiv.2110.14168 Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. https://doi.org/10.48550/arXiv.2110.14168 Training Verifiers to Solve Math Word Problems...

    doi arXiv

  5. [14]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021 b . Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  6. [15]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020 a . https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learni...

    2020 doi

  7. [16]

    Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020 b . https://doi.org/10.18653/v1/2020.acl-main.536 Emerging cross-lingual structure in pretrained language models . In Proceedings of the 58th Annual Meeting of the Association for Computational ...

    2020 doi

  8. [17]

    Marta R. Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip H...

    2024 doi

  9. [18]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  10. [19]

    Eberhard, Gary F

    David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2025. https://www.ethnologue.com Ethnologue: Languages of the World , 26 edition. SIL International, Dallas, Texas. Accessed July 3, 2025

    2025

  11. [20]

    Kawin Ethayarajh and Dan Jurafsky. 2020. https://aclanthology.org/2020.emnlp-main.272 Utility is in the eye of the user: A critique of nlp leaderboards . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3356--3363. Associ...

    2020

  12. [21]

    Jinlan Fu, See-Kiong Ng, and Pengfei Liu. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.674 Polyglot prompt: Multilingual multitask prompt training . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9919--9935, Abu Dhabi, Uni...

    2022 doi

  13. [22]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  14. [23]

    Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc ' Aurelio Ranzato, Francisco Guzm \'a n, and Angela Fan. 2022. https://doi.org/10.1162/tacl_a_00474 The F lores-101 evaluation benchmark for low-resource and multilingua...

    2022 doi

  15. [24]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  16. [25]

    Daniil Gurgurov, Rishu Kumar, and Simon Ostermann. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.67 G r E m LI n: A repository of green baseline embeddings for 87 low-resource languages injected with multilingual graph knowledge . In Findings of the Association for Com...

    2025 doi

  17. [26]

    https://doi.org/10.48550/arXiv.2009.03300 Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. https://doi.org/10.48550/arXiv.2009.03300 Measuring Massive Multitask Language Understanding . Preprint, arXiv:2009.03300

    2009 doi arXiv

  18. [27]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  19. [28]

    Rossi, and Thien Huu Nguyen

    Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. https://doi.org/10.48550/arXiv.2307.16039 Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human F...

    2023 doi arXiv

  20. [29]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  21. [30]

    Thomas Mayer and Michael Cysouw. 2014. https://aclanthology.org/L14-1215/ Creating a massively parallel B ible corpus . In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC `14) , pages 3158--3163, Reykjavik, Iceland. European Langua...

    2014

  22. [31]

    Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. https://arxiv.org/abs/2311.07978 Afrobench: How good are large language models on african languages? Preprint, arXiv:2311.07978

    2025 arXiv

  23. [32]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  24. [33]

    Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4876--4883

    2020

  25. [34]

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and S Yu Philip. 2025. A survey of multilingual large language models. Patterns, 6(1):101118

    2025

  26. [35]

    Inioluwa Deborah Raji, Andrew Smart, and Rumman White. 2021. https://doi.org/10.1145/3442188.3445944 Ai benchmarking: A meta-analysis and the need for inclusive, dynamic metrics . Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), page 1104–1114

    2021

  27. [36]

    Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 3118--3135

    2021

  28. [37]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. https://doi.org/10.48550/arXiv.2210.03057 Language Models are Multilingual Chain-of-Thought Reasoners...

    2022 doi arXiv

  29. [38]

    Shivalika Singh, Angelika Romanou, Cl \'e mentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero , Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut,...

    2025 doi arXiv

  30. [39]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786

    2025 arXiv

  31. [40]

    o hler, Ren \'e J \

    Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr , Joachim K \"o hler, Ren \'e J \"a kel, and Mehdi Ali. 2024 a . https://doi.org/10.48550/arXiv.2410.08928 Towards Multilingual LLM Eval...

    2024 doi arXiv

  32. [41]

    Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, and Mehdi Ali. 2024 b . https://arxiv.org/abs/2410.08928 Towards multilingual llm evaluation for european l...

    2024 arXiv

  33. [42]

    Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, and 1 others. 2024. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37...

    2024

  34. [43]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  35. [44]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  36. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  37. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.