Pith. sign in

REVIEW 3 major objections 6 minor 44 references

A continually pretrained 10B model, built from Llama-3.1-8B with only 80M Bavarian tokens, claims the top sub-10B Bavarian benchmark rank after instruction tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:14 UTC pith:FIJRYMAK

load-bearing objection A genuinely useful low-resource adaptation recipe whose headline Bavarian SOTA claim currently rests on an unvalidated, unreleased machine-translated benchmark — fixable, but not publishable as-is. the 3 major comments →

arxiv 2509.05668 v1 pith:FIJRYMAK submitted 2025-09-06 cs.CL cs.AI

Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian

Michael Hoffmann , Jophin John , Stefan Schweter , Gokul Ramakrishnan , Hoi-Fong Mak , Alice Zhang , Dmitry Gaynullin , Nicolay J. Hammer This is my paper
classification cs.CL cs.AI
keywords multilingual language modelcontinual pretrainingBavarian dialectlow-resource language modelingblock expansiontrilingual evaluationinstruction tuningEnglish-German-Bavarian
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a foundation model can serve English, German, and a low-resource dialect fairly without letting English dominate the training mix, and that even a very small dialect corpus can produce a state-of-the-art dialect model when it is introduced late in training. Starting from Llama-3.1-8B, the authors insert eight new Transformer blocks, continue training on 164B tokens (82B English, 82B German, 80M Bavarian), and then instruction-tune on a trilingual set of 867k prompts. Their central claim is that the fine-tuned model is the best sub-10B system on Bavarian benchmarks, ahead of gemma-2-9B-it and Apertus-8B-Instruct-2509, while matching EuroLLM-9B in German and beating it in English. If the claim holds, it matters because it offers a comparatively cheap recipe for adding an under-resourced dialect to an existing model, together with the first trilingual English-German-Bavarian benchmark suite. The evidence is accuracy on translated versions of ARC, HellaSwag, MMLU, TruthfulQA, and WinoGrande.

Core claim

The discovery the paper argues for is that Bavarian competence does not require a large Bavarian corpus. With Bavarian capped at 80M tokens—about 0.05% of the 164B-token continued-pretraining budget—the base model already lands near gemma-2-9b, Apertus-8B-2509, and Llama-3.1-8B on Bavarian tasks, and the instruction-tuned variant takes first place among all evaluated sub-10B models, outperforming gemma-2-9B-it and Apertus-8B-Instruct-2509. The paper also claims the same model beats EuroLLM-9B on English and matches it on German, suggesting the balanced 1:1 English-German mix did not cost English strength. These results are presented as evidence that block-expansion continual pretraining, wit

What carries the argument

The mechanism is a staged continual-pretraining recipe. Eight Transformer blocks are inserted into a frozen Llama-3.1-8B backbone, with the new attention-output and FFN-final layers zero-initialized, so the expanded 10B model preserves the base's knowledge while adding trainable capacity. For the first 90% of iterations the model sees only a 1:1 English-German mix, chosen after a 16B-token experiment showed that ratio beat a 9:1 German-heavy split; in the final 10%, Bavarian text is introduced, upsampled to 80M tokens. A tokenizer expanded by 20% with German and Bavarian subword units lowers token fertility. The paper credits this sequence—reserve new capacity, establish the high-resource la

Load-bearing premise

The headline Bavarian ranking stands on the assumption that the machine-translated Bavarian benchmark items measure Bavarian ability with the same difficulty and validity as the German originals, and the paper reports no human validation of those translations.

What would settle it

Back-translate the Bavarian versions of ARC/HellaSwag/MMLU/TruthfulQA/WinoGrande into German and compare with the originals; if many items change their correct answer, leak the answer through translationese, or prove much easier for native speakers than the German originals, the claimed Bavarian superiority may be an artifact of the test rather than the model.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A dialect can become a model's best language with a token share below 0.1%, if the dialect data arrive after the high-resource backbone is stable.
  • A 1:1 English-German continued-pretraining mix supports strong English and German results simultaneously, so English dominance is not required for a capable European model.
  • Instruction tuning on machine-translated English/German instruction data is enough to convert a competitive base model into the leading sub-10B Bavarian system.
  • The 35.23 MWh / 66-day run on a single Cerebras CS-2 offers a concrete energy baseline for small teams attempting similar multilingual continual pretraining.
  • The translated ARC/HellaSwag/MMLU/TruthfulQA/WinoGrande suite becomes a reusable trilingual benchmark for future German-English-Bavarian systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the staging result suggests the timing of low-resource data exposure may matter more than its raw volume; an ablation that feeds Bavarian throughout training, or doubles the late-stage Bavarian budget, would test that directly.
  • Editorial inference: the same 'translate a high-resource benchmark, then reuse it' template could be applied to other German dialects, but each new test set would need its own native-speaker validation before its leaderboard is trusted.
  • Editorial inference: the energy and wall-clock numbers are specific to the CS-2 hardware, so the 'efficient for small teams' message should be read as a hardware-dependent data point rather than a universal cost guarantee.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Llama-GENBA-10B, a 10B-parameter trilingual model obtained by block-expansion continual pretraining of Llama-3.1-8B on 164B tokens (82B English, 82B German, 80M Bavarian). The development pipeline includes corpus construction, a trilingual tokenizer extension, a language-ratio experiment, staged introduction of Bavarian data, and supervised fine-tuning on 867k instruction pairs. The central claims are that the base model is competitive in English and German and strong in Bavarian, and that the instruction-tuned variant is the best sub-10B model on Bavarian, surpassing Apertus-8B-Instruct-2509 and gemma-2-9B-it. Evaluation is performed with English and German versions of ARC, HellaSwag, TruthfulQA, MMLU, and WinoGrande, plus Bavarian versions obtained by translating the German benchmarks. The paper also reports energy consumption for pretraining on a Cerebras CS-2.

Significance. If the Bavarian benchmark were validated and released, the paper would offer a useful, resource-efficient blueprint for adding a low-resource dialect to an existing LLM, and the documented energy measurement is a concrete contribution. The 1:1 English/German ratio experiments and the staged Bavarian data integration are informative. However, the headline result is not currently establishable: the Bavarian evaluation instrument is described in one truncated sentence, is not released, and is not validated against human judgments or existing Bavarian resources. The central SOTA claim is therefore conditional on an unverified benchmark.

major comments (3)
  1. [§5.1 (Bavarian benchmark) and §5.2 (headline claim)] The central claim that Llama-GENBA-10B-instruct is the best sub-10B Bavarian model rests entirely on the Bavarian benchmark introduced in §5.1. That section contains only the truncated sentence "For Bavarian we translated the German versions of these benchmarks to enable evaluation in" and provides no translation system, prompt, human validation, back-translation check, or release of the test set. This is load-bearing because §4.4 states that all Bavarian post-training data were produced by automatic translation with Gemini-flash; if the benchmark translations also use Gemini-flash, the fine-tuned model is evaluated on the same translationese it was trained on, which biases the comparison against baselines that were not adapted to that translationese. An unreleased test set also cannot be checked for overlap with the 867k-pair fine-tuning corpus. Until the benchmark is released and valid
  2. [§5.1, Table 6] All comparisons are reported as single point estimates with no error bars, repeated runs, or significance tests. In Table 6, the ARC difference between Llama-GENBA-10B-base (0.3776) and EuroLLM-9B (0.3771) is 0.0005, yet the text describes the model as outperforming EuroLLM-9B. With a single evaluation run, such differences are within evaluation noise. Please provide confidence intervals, multiple seeds, or at least standard errors for the key comparisons, and report Bavarian results per benchmark rather than only through aggregate figures.
  3. [§4.1, Table 3] The 1:1 English/German language-mix ratio is selected using small 16B-token experiments reported in Table 3. The paper does not state which evaluation tasks are used for the Acc-EN/Acc-DE numbers. If these are the same ARC/HellaSwag-style tasks that later define success in §5, then the 1:1 choice is a model-selection step on the evaluation metric, making the later "outperforms EuroLLM" comparisons less clean. Please either specify that a separate held-out set was used, or acknowledge the selection effect and report the evaluation on a distinct, pre-registered test set.
minor comments (6)
  1. [§5.1] The sentence describing Bavarian benchmark construction is truncated: "For Bavarian we translated the German versions of these benchmarks to enable evaluation in". The missing text must be restored, and the translation methodology must be fully specified.
  2. [Table 6] Table 6 does not indicate which language the scores refer to. The text reports results separately for English, German, and Bavarian, but the table appears to present only a single set of numbers. Please provide separate tables or clearly labeled language columns.
  3. [Figures 3 and 4] The bar charts are difficult to read and do not show numeric values or uncertainty. Add numeric labels and, where available, error bars so readers can verify the claims in the text.
  4. [§3.2] The "fertility score" is used to select the tokenizer expansion but is never defined. Please provide the formula and also report downstream task performance for the candidate tokenizers, since fertility alone does not determine model quality.
  5. [§6] The conclusion states that the base model achieves "state-of-the-art performance in Bavarian," while §5.1 reports that it ranks fourth among baselines in Bavarian. This inconsistency should be corrected.
  6. [References] Some references are incomplete or appear mislabeled (e.g., the Eurostat 2024 citation is listed under Scientific Reports). Please verify all entries.

Circularity Check

0 steps flagged

No significant circularity; the main evaluation claims rest on external benchmarks, and the self-cited block-expansion method is not load-bearing.

full rationale

The paper's load-bearing claim is that Llama-GENBA-10B-instruct is the best sub-10B model on Bavarian. That comparison is made on a benchmark suite assembled by the authors from existing English/German benchmarks plus a Bavarian translation introduced in §5.1. While this makes the benchmark a proprietary instrument, the claim does not reduce by construction to the training data or to a fitted parameter: the model's scores are measured, not derived from the benchmark definition. The language-ratio experiment in §4.1 selects a 1:1 English/German mix using the same family of tasks (ARC/HellaSwag etc.) that later appears in the evaluation, but this is transparent model selection over a 16B-token proxy, not a fitted parameter that is later renamed a prediction; the final model trains on 164B tokens and is compared against independent external baselines. The block-expansion recipe is attributed to both Wu et al. (2024) and Gosal et al. (2024); the latter shares an author (G. Ramakrishnan), but the method is also supported by an external citation and is not used to establish the evaluative results. The Bavarian benchmark translation procedure is incompletely described (§5.1: 'For Bavarian we translated the German versions of these benchmarks to enable evaluation in' — sentence truncated), and the fine-tuning data were produced with Gemini-flash translations (§4.4). If the benchmark translations used the same translator, this would be a serious validity/leakage confound, but the paper does not state that, and a possible confound is not a circular reduction. No equation, parameter, or benchmark score is shown to be equivalent to its own input. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central claims rest on choices made by hand (token ratio, upsampling, tokenizer size, timing of Bavarian insertion) plus the unvalidated assumption that machine translation produces sound Bavarian benchmarks. No new physical entities are introduced; the only new artifacts are datasets and a benchmark, which are captured as free parameters and assumptions.

free parameters (4)
  • English:German token ratio = 1:1 (8B/8B in probe)
    Selected because scenario A (1:1) scored Acc-EN 0.4040, Acc-DE 0.5256 vs scenario B (9:1) 0.4009/0.5169 (Table 3). This ratio shapes the whole pretraining run and was chosen on the same benchmarks used for final evaluation.
  • Bavarian upsampling factor = 4x (20M to 80M tokens)
    Table 1 lists a 20M-token Bavarian corpus; Table 2 trains on 80M tokens, i.e., a 4x upsample. The factor is not justified by experiment.
  • Tokenizer vocabulary expansion = 20% (+25,600 tokens)
    Chosen from three variants (10/20/30%) by diminishing fertility-score returns (1.9026 to 1.8372 to 1.8214), a heuristic criterion tied to the tokenizer's own design.
  • Bavarian introduction timing = final 10% of training
    Bavarian data enter only after 90% of iterations. Motivated qualitatively (reduced interference), not supported by an ablation.
axioms (3)
  • domain assumption Machine translation with Gemini-flash yields valid Bavarian benchmark items
    Section 5.1 and 4.4: German benchmarks and instruction pairs are translated into Bavarian with no human validation, back-translation check, or dialect acceptance test. The headline Bavarian SOTA claim depends on this.
  • domain assumption GlotLID-m classification of FineWeb yields genuine Bavarian sentences
    Section 3.1: the Bavarian corpus relies on ~262k lines extracted by GlotLID-m after FastText proved inadequate; correctness of the dialect filter is unverified beyond the classifier's confidence.
  • domain assumption Zero-initialized inserted blocks preserve learned behavior before gradient updates
    Section 4.2: the block-expansion recipe is imported from Gosal et al. 2024 and Wu et al. 2024; the paper provides no ablation isolating the contribution of zero-init plus frozen backbone in the trilingual setting.

pith-pipeline@v1.3.0-alltime-deepseek · 13201 in / 16130 out tokens · 148709 ms · 2026-08-05T05:14:02.302870+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian." pith.science (2026). https://pith.science/paper/FIJRYMAK

@misc{pith2026250905668,
  author       = {Pith},
  title        = {Pith review of: Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FIJRYMAK}},
  note         = {Machine review of arXiv:2509.05668}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present Llama-GENBA-10B, a trilingual foundation model addressing English-centric bias in large language models. Built on Llama 3.1-8B and scaled to 10B parameters, Llama-GENBA-10B is continuously pretrained on 164B tokens (82B English, 82B German, and 80M Bavarian), balancing resources while preventing English dominance. Targeted at the German NLP community, the model also promotes Bavarian as a low-resource language. Development tackled four challenges: (1) curating a multilingual corpus despite Bavarian scarcity, (2) creating a unified tokenizer for English, German, and Bavarian, (3) optimizing architecture and language-ratio hyperparameters for cross-lingual transfer, and (4) establishing the first standardized trilingual evaluation suite by translating German benchmarks into Bavarian. Evaluations show that Llama-GENBA-10B achieves strong cross-lingual performance, with the fine-tuned variant surpassing Apertus-8B-2509 and gemma-2-9b in Bavarian and establishing itself as the best model in its class for this language, while also outperforming EuroLLM in English and matching its results in German. Training on the Cerebras CS-2 demonstrated efficient large-scale multilingual pretraining with documented energy use, offering a blueprint for inclusive foundation models that integrate low-resource languages.

Figures

Figures reproduced from arXiv: 2509.05668 by Alice Zhang, Dmitry Gaynullin, Gokul Ramakrishnan, Hoi-Fong Mak, Jophin John, Michael Hoffmann, Nicolay J. Hammer, Stefan Schweter.

Figure 1
Figure 1. Figure 1: Comparative Performance Across Evaluation [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance For Different Language Splits [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of Llama-GENBA-10B-base against peer base models, showing competitive performance [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison of Llama-GENBA-10B-instruct against peer instruction-tuned models, showing [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 14 canonical work pages · 7 internal anchors

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  4. [4]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max L \"u bbering, Daniel Steinigen, Johannes Leveling, et al. 2024 a . Teuken-7b-base & teuken-7b-instruct: Towards european llms. arXiv preprint arXiv:2410.03730

    Pith/arXiv arXiv 2024

  5. [5]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L \"u bbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, et al. 2024 b . Tokenizer choice for llm training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907--3924

    work page 2024

  6. [6]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. http://arxiv.org/abs/2310.10631 Llemma: An open language model for mathematics

    Pith/arXiv arXiv 2023

  7. [7]

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. https://huggingface.co/datasets/HuggingFaceTB/cosmopedia Cosmopedia

    2024

  8. [8]

    Verena Blaschke, Barbara Kova c i \'c , Siyao Peng, Hinrich Sch \"u tze, and Barbara Plank. 2024. Maibaam: A multi-dialectal bavarian universal dependency treebank. arXiv preprint arXiv:2403.10293

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Verena Blaschke, Miriam Winkler, Constantin F \"o rster, Gabriele Wenger-Glemser, and Barbara Plank. 2025. A multi-dialectal dataset for german dialect asr and dialect-to-standard speech translation. arXiv preprint arXiv:2506.02894

    work page arXiv 2025

  10. [10]

    Branden Chan, Stefan Schweter, and Timo M \"o ller. 2020. German's next language model. arXiv preprint arXiv:2010.10906

    Pith/arXiv arXiv 2020

  11. [11]

    Monojit Choudhury, Shivam Chauhan, Rocktim Jyoti Das, Dhruv Sahnan, Xudong Han, Haonan Li, Aaryamonvikram Singh, Alok Anil Jadhav, Utkarsh Agarwal, Mukund Choudhary, et al. 2025. Llama-3-nanda-10b-chat: An open generative large language model for hindi. arXiv preprint arXiv:2504.06011

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  13. [13]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261

    Pith/arXiv arXiv 2025

  14. [14]

    Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv e-prints, pages arXiv--2307

    work page 2023

  15. [15]

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366--3385

    work page 2021

  16. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  17. [17]

    Eurostat. 2024. Electricity and heat statistics. Scientific Reports

    work page 2024

  18. [18]

    Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Conghui He, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Query of cc: unearthing large scale domain-specific knowledge from public corpora. arXiv preprint arXiv:2401.14624

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    Pablo Gamallo, Pablo Rodr \' guez, Daniel Santos, Susana Sotelo, Nuno Miquelina, Silvia Paniagua, Daniela Schmidt, Iria de Dios-Flores, Paulo Quaresma, Daniel Bardanca, et al. 2024. A galician-portuguese generative model. In EPIA Conference on Artificial Intelligence, pages 292--304. Springer

    work page 2024

  20. [20]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. https...

    doi:10.5281/zenodo.12608602 2024

  21. [21]

    Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Biswajit Mishra, Natalia Vassilieva, Joel Hestness, Neha Sengupta, Sunil Kumar Sahu, et al. 2024. Bilingual adaptation of monolingual foundation models. arXiv preprint arXiv:2407.12869

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    Pith/arXiv arXiv 2020

  23. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \'e lio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth \'e e Lacroix, and William El Sayed. 2023. Mistral 7b. arXiv pr...

    Pith/arXiv arXiv 2023

  24. [24]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759

    Pith/arXiv arXiv 2016

  25. [25]

    Amir Hossein Kargaran, Ayyoob Imani, Fran c ois Yvon, and Hinrich Sch \"u tze. 2023. Glotlid: Language identification for low-resource languages. arXiv preprint arXiv:2310.16248

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    Wen Lai, Mohsen Mesgar, and Alexander Fraser. 2024. Llms beyond english: Scaling the multilingual capability of llms with cross-lingual feedback. arXiv preprint arXiv:2406.01771

    Pith/arXiv arXiv 2024

  27. [27]

    Sean Lie. 2024. https://doi.org/10.1109/MM.2024.3386628 Inside the cerebras wafer-scale cluster . IEEE Micro, 44(3):49--57

    work page arXiv 2024

  28. [28]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958

    Pith/arXiv arXiv 2021

  29. [29]

    Pedro Henrique Martins, Jo \ a o Alves, Patrick Fernandes, Nuno M Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M Alves, Jos \'e Pombal, Manuel Faysse, et al. 2025. Eurollm-9b: Technical report. arXiv preprint arXiv:2506.04079

    Pith/arXiv arXiv 2025

  30. [30]

    Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. arXiv preprint arXiv:2205.06266

    Pith/arXiv arXiv 2022

  31. [31]

    Jan Pfister, Julia Wunderle, and Andreas Hotho. 2024. Ll " ammlein: Compact and competitive german-only language models from scratch. arXiv preprint arXiv:2411.11171

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Translationese as a Language in "Multilingual" NMT

    Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2019. Translationese as a language in" multilingual" nmt. arXiv preprint arXiv:1911.03823

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  33. [33]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. https://doi.org/10.1145/3474381 Winogrande: an adversarial winograd schema challenge at scale . Commun. ACM, 64(9):99–106

    doi:10.1145/3474381 2021

  34. [34]

    Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149

    Pith/arXiv arXiv 2023

  35. [35]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    Pith/arXiv arXiv 2023

  36. [36]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    Pith/arXiv arXiv 2023

  37. [37]

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D ' souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. https://doi.org/10.18653/v1/2024.acl-long.845 Aya model: An instruction finetuned ...

    doi:10.18653/v1/2024.acl-long.845 2024

  38. [38]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  39. [39]

    Zirui Wang, Zachary C Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual models: Findings and a meta-learning treatment. arXiv preprint arXiv:2010.03017

    Pith/arXiv arXiv 2020

  40. [40]

    Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018

    Pith/arXiv arXiv 2023

  41. [41]

    BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100

    Pith/arXiv arXiv 2022

  42. [42]

    Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. 2024. Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415

    Pith/arXiv arXiv 2024

  43. [43]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  44. [44]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    Pith/arXiv arXiv 2019