Pith. sign in

REVIEW 3 major objections 6 minor 46 references

A 2.7B German-first language model, trained on public data with German-specific filtering and educational rephrasing, is the strongest under 3B and matches 7B models on German benchmarks while running on phones.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 11:13 UTC pith:X442XOSF

load-bearing objection Solid German-first systems paper with real ablations and a usable 2.7B checkpoint; the “best <3B / matches 7B” headline is narrower and less controlled than it sounds, mainly because of missing decontamination and thin eval. the 3 major comments →

arxiv 2607.24585 v1 pith:X442XOSF submitted 2026-07-27 cs.CL

From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference

Darina Gold , Alexander Schwirjow , Viktor Haag , Viktor Hangya , Joel Schlotthauer , Fabian K\"uch , Luzian Hahn This is my paper
classification cs.CL
keywords German language modelon-device inferencedata filteringeducational quality scoringsynthetic rephrasingtokenizer designlearning-rate annealingmobile LLM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper shows how to build a compact German-capable language model from scratch under a tight compute budget and still get strong German benchmark scores. The authors argue that English-oriented web filtering is a poor fit for German morphology, compounding, and orthography, so they rebuild the pre-processing stack and add an educational-quality scorer plus targeted rephrasing of mid-quality text. With those data choices, a polynomial learning-rate schedule with an annealing phase, and a tokenizer tuned for German–English–code mix, their 2.7B model becomes the best in its size class and reaches the level of much larger 7B systems on German ARC, HellaSwag, and MMLU. The practical point is that careful data work can substitute for raw scale when the goal is local, private, on-device inference rather than a remote megamodel.

Core claim

ELMOD-2.7B, trained on about 4T tokens with 55k H100 GPU hours of exclusively public data, is the strongest German-capable base model below 3B parameters and matches the performance of 7B-parameter models on German versions of ARC, HellaSwag, and MMLU. The gain is attributed to German-adapted rule and gazetteer filters, retention of documents scored FWe-de ≥ 2 by an educational-quality classifier, rephrasing of near-threshold text into textbook- or blog-style form, a 65k three-digit tokenizer with a 50/40/10 English–German–code mix, and polynomial learning-rate annealing on the final high-quality tokens.

What carries the argument

FineWebEdu German (FWe-de): an educational-quality score obtained by labeling web documents with an LLM judge on a 0–5 scale, then training a linear regressor on text embeddings; documents at FWe-de ≥ 2 are kept for pre-training, and mid-bucket (1.5–2) German text is rephrased into higher-educational-quality forms before the annealing phase.

Load-bearing premise

The educational-quality score is a valid proxy for what actually improves German downstream ability under a fixed token budget, so filtering and rephrasing truly raise learning efficiency rather than just reshaping the data toward the evaluation style.

What would settle it

Train an otherwise identical 2.7B model on the same token budget using only the unfiltered post-rule-filter German crawl (no FWe-de threshold and no rephrasing) and check whether it still matches or beats ELMOD on German ARC, HellaSwag, and MMLU after the same annealing horizon.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • German on-device assistants can reach 7B-class benchmark quality without 7B-class memory or server dependence.
  • Mid-quality web text can be converted into annealing fuel by style-controlled rephrasing instead of being discarded.
  • Tokenizer digit handling, vocabulary size, and language mix should be ablated for the target language, not inherited from English defaults.
  • A polynomial schedule with a short high-quality anneal is preferable when total pre-training tokens are uncertain.
  • The same German-first filtering and educational-scoring pipeline can be reused as a template for other non-English on-device models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If educational rephrasing mainly compresses away ads and boilerplate, similar gains may appear in other morphologically rich languages without needing a full new architecture search.
  • The gap between strong base-model ranking and only middling instruction-tuned ranking suggests post-training data, not pre-training, is now the binding constraint for this size class.
  • Publishing the FWe-de regressor and German filter configs would let others test whether the efficiency claim replicates under different judges or embedding backbones.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents ELMOD-2.7B, a German-first 2.7B-parameter base model trained from scratch on 4T tokens (45% German / 45% English / 10% code) within a stated budget of 55k H100 GPU hours, plus a minimal instruction-tuned variant and an on-device Android deployment. The main contributions are a German-adapted heuristic filtering chain (language, line, document, URL, profanity, Bloom/MinHash dedup), an educational-quality regressor (FWe-de) distilled from Mistral-Small-24B judgments on 1.4M documents, rephrasal of FWe-de(1.5,2) documents into textbook/blog styles with Qwen3-8B (73B synthetic tokens), regex-based anonymization, a 66-configuration tokenizer study, and a polynomial LR schedule with annealing selected via proxy grid search. The headline claim is that ELMOD-2.7B is the strongest German-capable base model under 3B parameters and matches 7B models on German ARC, HellaSwag, and MMLU (Thellmann et al., 2024 translations), evaluated in cloze formulation via lm-eval-harness.

Significance. If the comparative results hold up, this is a useful reference for the German/European LM community: an end-to-end, compute-budgeted recipe for a non-English small model with unusually complete disclosure — concrete filter configurations (Listings 2–3), deduplication settings (Table 4), LLM-judge and rephrasal prompts (Listings 4–7), tokenizer pre-tokenizer regex (Listing 1), and a released HF model collection. The ablation program (six quality-bucket runs in Fig. 2, a 66-configuration tokenizer grid in Fig. 7, four LR/annealing variants in Fig. 3, a 480-run proxy grid for schedule parameters) is extensive for a 55k-GPU-hour project and gives the design choices empirical support rather than asserting them. Table 1's cross-device throughput measurements make the on-device deployment claim concrete and falsifiable.

major comments (3)
  1. [§3.2, §3.4, Figs. 4–5] The load-bearing claim ('strongest performer <3B, matching 7B models') rests entirely on German ARC/HellaSwag/MMLU (Thellmann et al., 2024), yet §3.2 documents no benchmark decontamination step. Those German translations and associated artifacts were released in late 2024; the German web data extends to the March 2025 CC snapshot, so leaderboard scrapes, dataset cards, and eval-harness repos quoting items can survive the heuristic filters, and the FWe-de≥2 criterion does not target eval leakage. §3.4 adds a second vector: the rephraser Qwen3-8B has almost certainly seen the English originals of these suites. This is a correctness risk, not an assumption of misconduct: please add an n-gram overlap audit of the 4T-token corpus (and the 73B rephrased tokens) against the three German suites, and/or re-evaluate on a decontaminated split, and report the result.
  2. [§4.3, Figs. 4–5] The headline comparison is presented only as bar charts; no numerical scores appear in text or tables, there is no per-benchmark breakdown (only the three-task average), and all results are single-seed with no variance estimate. At 2.7B scale on cloze-formulation MCQ, run-to-run and harness noise is of the order of a point, so the (unreported) margin over Llama-3.2-3B-class baselines cannot be assessed. Please provide a results table with per-task German scores for ELMOD-2.7B and all comparison models, state the evaluation protocol (shots, cloze formulation, harness version) per model, and give either multi-seed runs or an estimate of evaluation noise so that 'outperforms all <3B models' and 'matches 7B' can be verified.
  3. [Abstract, §4.2–4.3] The abstract and §1 headline 'limited computational budget (55k H100 GPU hours)', but a back-of-envelope check (6ND at plausible MFU) suggests the 4T-token production run alone consumes roughly this budget, leaving the 480-run LR grid search, 66 tokenizer models, six 20B-token quality-bucket ablations, four 200B-token LR runs, the 1.4M-document Mistral-Small-24B judging, and the 73B-token Qwen3-8B rephrasal unaccounted for. Please clarify the scope of the 55k figure (final run only vs. end-to-end) and add a compute-accounting table; this matters because compute efficiency is one of the paper's stated contributions.
minor comments (6)
  1. [Figs. 4–5] Cross-referencing error: the text attributes the Llämmlein-7B comparison to Figure 5, whose caption restricts the comparison to 'models <= 3' (presumably ≤3B parameters). Figure 4's caption says models 'was trained with similar amounts of pretraining tokens' while the §4.3 text and Figure 5 caption say ELMOD used 'much less tokens'. Units are missing in both captions ('<= 3are hatched', '<= 3parameters').
  2. [§4.1, Fig. 7] The tokenizer conclusions are stated more strongly than the evidence supports: Fig. 7(a)'s y-axis spans 0.262–0.274 and the digit-handling and vocabulary curves differ by ~0.002–0.005 average agreement with no variance shown, yet the text says 3-digit 'clearly outperforms'. Please soften the wording or add spread estimates. The data-mix conclusion is appropriately hedged ('top configurations are similar').
  3. [§1, §3] First-of-kind claims should be tempered: 'first to outline the model creation process ... for a German model' (§1) and 'first to explicitly describe [anonymization] in a model training pipeline' (§3) sit uneasily next to Llämmlein (Pfister et al., 2025) and Aleph-Alpha-GermanWeb (Burns et al., 2026), which also document German pipelines end-to-end. A sentence differentiating the specific novelty (on-device focus, budget, full config release) would be more defensible.
  4. [Various] Typos/formatting: 'perform measurable better' (§3.3); 'on difference devices' (Table 1 caption); missing spaces in 'Mistral-Small-24B-Instruct-2501to' and 'onsnowflake-arctic-embed-mtext' (§3.3); 'Appendix 4' should presumably be 'Listing 4 / Appendix D' (§3.3); Table 4 uses periods as thousands separators (2.205.239), ambiguous for international readers; Table 3's 'F W e-de' is broken formatting.
  5. [References (Plüster, 2023)] The LeoLM reference is a blog URL containing 'utm_source=chatgpt.com'; please cite a stable source and clean the tracking parameter.
  6. [Appendix, Table 2] Table 2's single-dump proxy for filtering yield is reasonably acknowledged, but since 107 dumps were processed, reporting the yield range across a handful of dumps (even 3–5) would strengthen the representativeness claim at little cost.

Circularity Check

1 steps flagged

Empirical systems paper with external benchmarks; only minor non-load-bearing self-citation for architecture defaults.

specific steps
  1. self citation load bearing [§4 Architecture paragraph; also Abstract/Intro ‘architectural model and data choices’]
    "In our previous study (Schlotthauer et al., 2025), we investigated how to pre-train small (∼3 B parameter) language models under a fixed compute budget and concluded that AdamW consistently yields the strongest downstream task performance, making it the best all-around choice under limited pre-training resources. ... The final architecture—32 layers, 32 attention heads (with grouped-query attention for memory efficiency), head dimension 80, embedding size 2560, LayerNorm, no bias terms, a 2-layer MLP, and GELU activation was chosen for compute-efficient competitiveness. ELMOD-2.7b’s architectu"

    Architecture and optimizer defaults are taken from prior work by overlapping authors rather than re-derived here. This is ordinary incremental self-citation, not definitional circularity: it does not define the educational filter, does not force the German benchmark scores, and the paper independently ablates tokenizer, data mix, and annealing. Listed only as a minor non-load-bearing self-reference; central performance claim remains externally grounded.

full rationale

ELMOD’s central claim is comparative and empirical (best <3B / matches 7B on German ARC, HellaSwag, MMLU), not a first-principles derivation. Training data are public crawls; quality labels come from an independent judge (Mistral-Small-24B) plus a regressor on frozen embeddings, then validated by held-out ablations (Fig. 2) rather than defined as the ELMOD loss. Evaluation uses external standard suites and independent comparison models. Tokenizer, data-mix, and LR-annealing choices are ablated in-paper. The only self-reference is adoption of architecture/optimizer defaults from Schlotthauer et al. (2025) (overlapping authors); that citation is not used to forbid alternatives or to force the headline performance claim, and the authors explicitly deviate on the learning-rate schedule. No fitted parameter is renamed a prediction, no uniqueness theorem is imported, and no known result is merely relabeled. Contamination risk (if any) is a correctness issue, not circularity. Score 1 only for the minor architecture self-citation.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central performance claim rests on standard LM-training practice plus several tunable thresholds and the unproven but empirically motivated premise that educational-quality scores transfer to German downstream gains. No new physical entities; free parameters are engineering knobs (thresholds, mix ratios, schedule exponents) chosen by ablation or hand.

free parameters (6)
  • FWe-de retention threshold = ≥2 (from proxy ablations)
    Documents kept at FWe-de≥2 after comparing buckets 0/1/2/3 on 20B-token proxy runs; drives final pretrain mix and annealing data.
  • Profanity ratio threshold = 0.005
    Document dropped if profane-word ratio exceeds hand-set cutoff using LDNOOBW lists.
  • MinHash dedup configuration = threshold 0.8; 128/9×13
    Similarity threshold and banding (128 perms, 9 buckets × 13 rows) chosen for cost/quality tradeoff following FineWeb-style practice.
  • Polynomial LR schedule parameters a, b = optimized on proxies (values not numerically listed in text)
    Power-law decay parameters selected via 480-run grid on proxy models then transferred to 2.7B.
  • Pretrain data mix and tokenizer vocab = 50/40/10; vocab 65K; 3-digit
    Final 50% EN / 40% DE / 10% code and 65K vocab with 3-digit left-to-right handling chosen from tokenizer grid; production mix stated as 45/45/10 at data-collection stage then refined.
  • Annealing horizon split = 180B + 20B anneal
    Stable phase 180B tokens then linear anneal on last 20B; horizon chosen under uncertain data volume.
axioms (5)
  • domain assumption Educational-quality scores from an English-centric judge model plus embedding regression predict German LM sample efficiency.
    §3.3 imports FineWeb-Edu / Su / Burns methodology; validity for German is supported only by the paper’s own 20B ablations, not by external theory.
  • domain assumption German cloze-form ARC, HellaSwag, and MMLU (Thellmann et al., 2024) are adequate proxies for German language capability of small base models.
    §4 evaluation protocol; authors cite Gu et al. for cloze formulation but do not validate coverage of morphology, compounding, or generative German use.
  • domain assumption AdamW + µP-scaled ~3B Transformer with GQA is near-optimal under the fixed compute budget.
    Architecture taken from authors’ prior optimizer study (Schlotthauer et al., 2025) without re-sweeping full architecture space here.
  • domain assumption Standard web-filter heuristics (C4/Gopher-like) remain valid after German stopword, length, and keyword translation adjustments.
    §3.2; adjustments are described but not proven optimal beyond yield and downstream proxy gains.
  • standard math Byte-pair encoding with NFKC and accent stripping is an acceptable tokenizer inductive bias for German.
    Standard subword modeling assumptions; accent stripping is a deliberate design choice that may hurt some German distinctions.
invented entities (2)
  • FWe-de (FineWebEdu German) quality regressor no independent evidence
    purpose: Score every pretraining document for educational value to filter and select rephrase candidates.
    New trained artifact (linear regression on arctic-embed-m from Mistral-Small labels); independent evidence is only internal ablations, not external benchmarks of the scorer itself.
  • ELMOD-2.7B model family independent evidence
    purpose: The delivered base and instruction-tuned checkpoints for German-capable on-device inference.
    Primary artifact; evidence is the reported benchmark tables and device TPS measurements, not a new theoretical object.

pith-pipeline@v1.2.0-grok45-kimik3 · 23111 in / 4010 out tokens · 96330 ms · 2026-07-31T11:13:27.975238+00:00 · methodology

0 comments
read the original abstract

We present ELMOD - Efficient Language Model for On-Device Deployment - a compact (2.7B) German language model designed for efficient inference on resource-constrained hardware. ELMOD was trained on a limited computational budget (55k H100 GPU hours) using exclusively publicly available data. We developed a suite of German-specific data pre-processing, which differ from English-oriented counterparts in their handling of morphological variation, compounding, and orthographic conventions. Furthermore, we introduced a quality filtering and rephrasing step, which increased the instructional quality of the data, improved performance during the annealing phase, and reduced overall compute requirements. Thanks to our architectural model and data choices, including prefiltering, our educational-quality filtering and rephrasal to raise the educational-quality, ELMOD is the strongest performer in its size class (<3B), matching the performance of 7B-parameter models in German.

Figures

Figures reproduced from arXiv: 2607.24585 by Alexander Schwirjow, Darina Gold, Fabian K\"uch, Joel Schlotthauer, Luzian Hahn, Viktor Haag, Viktor Hangya.

Figure 1
Figure 1. Figure 1: Overview over model creation process Common Crawl (CC) CC3 is one of the largest publicly available web-scale text datasets, con￾taining hundreds of terabytes of raw web data across many languages. Additionally, CC offers high diversity across domains—ranging from news, blogs, and scientific articles to forums, reviews, and Wikipedia—helping reduce domain bias and improve model robustness. Finally, it has … view at source ↗
Figure 2
Figure 2. Figure 2: Average model performance on data of vary [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average performance of learning rate strate [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of German-capable base mod￾els with 1B–8B parameters, averaged across German base tasks; models <= 3 are hatched; ELMOD-2.7B performs on par with 7B models and was trained with similar amounts of pretraining tokens. * marks missing information on number of pretraining tokens [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Evaluation results of our instruction-tuned [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of different tokenizer con￾figurations averaged over all tasks (ARC, Hel￾laSwag, MMLU), with gpt-2 as a reference a) shows different digit handling: 3-DIGITS clearly outperforms 1-DIGITS; b) shows different vocabu￾lary size configurations: 65K (65024VOCAB), 52K (52096VOCAB), and 16K (16000VOCAB). While 16K performs worst, 52K is only slightly worse than 65K) c) different data ratio configuration… view at source ↗
Figure 8
Figure 8. Figure 8: Exemplary screenshot of our Android demo [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Average performance of different learning rate [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 3 canonical work pages

  1. [1]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat, Hammam Abdelwahab, Benny Jörg Stein, Karl-Heinz Sylla, Pavel Denisov, Nicolo' Brandizzi, Qasid Saleem, and 22 others. 2025. https://arxiv.org/...

    Pith/arXiv arXiv 2025

  2. [2]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L \"u bbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, and 2 others. 2024. https://doi.org/10.18653/v1/2024.fin...

    doi:10.18653/v1/2024.findings-naacl.247 2024

  3. [3]

    Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2018. https://link.springer.com/chapter/10.1007/978-3-319-76941-7_83 Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl . In Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018), Lecture Notes in Computer Science, Berlin Heidelberg New Y...

    work page doi:10.1007/978-3-319-76941-7_83 2018

  4. [4]

    Burton H Bloom. 1970. https://dl.acm.org/doi/abs/10.1145/362686.362692 Space/time trade-offs in hash coding with allowable errors . Communications of the ACM, 13(7):422--426

    arXiv 1970

  5. [5]

    Andrei Z. Broder. 1997. https://ieeexplore.ieee.org/abstract/document/666900 On the resemblance and containment of documents . In Proceedings of the Compression and Complexity of Sequences 1997 (SEQUENCES '97), pages 21--29. IEEE

    1997

  6. [6]

    Thomas F Burns, Letitia Parcalabescu, Stephan Waeldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren, and Bj \"o rn Deiseroth. 2026. https://doi.org/10.18653/v1/2026.eacl-long.58 Aleph-alpha- G erman W eb: Improving G erman-language LLM pre-training with model-based data curation and synthetic data generation . In Proceedings of the 19...

    work page doi:10.18653/v1/2026.eacl-long.58 2026

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/pdf/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . arXiv:1803.05457v1

    Pith/arXiv arXiv 2018

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/pdf/2110.14168 Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168

    Pith/arXiv arXiv 2021

  9. [9]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2020.acl-main.747 2020

  10. [10]

    Pieter Delobelle and Alan Akbik. 2024. https://huggingface.co/flair/bueble-lm-2b BübleLM: A small German LM

    2024

  11. [11]

    Jesse Dodge, Maarten Sap, Ana Marasovi \'c , William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.98 Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process...

    doi:10.18653/v1/2021.emnlp-main.98 2021

  12. [12]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. https://doi.org/10.5281/zenodo.12608602 The languag...

    doi:10.5281/zenodo.12608602 2024

  13. [13]

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.282 OLMES : A standard for language model evaluations . In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5005--5033, Albuquerque, New Mexico. Association for Computational Linguistics

    doi:10.18653/v1/2025.findings-naacl.282 2025

  14. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. https://arxiv.org/pdf/2009.03300 Measuring massive multitask language understanding . arXiv preprint arXiv:2009.03300

    Pith/arXiv arXiv 2020

  15. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. https://arxiv.org/abs/2310.0...

    Pith/arXiv arXiv 2023

  16. [16]

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. https://doi.org/10.18653/v1/2022.acl-long.577 Deduplicating training data makes language models better . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424...

    doi:10.18653/v1/2022.acl-long.577 2022

  17. [17]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, and 40 others. 2024. https://doi.org/10.52202/079017-0455 Datacomp-lm: In search of the ...

    doi:10.52202/079017-0455 2024

  18. [18]

    Zihao Li, Shaoxiong Ji, Hengyu Luo, and J \"o rg Tiedemann. 2025. https://openreview.net/forum?id=mpTIzK4Zca Rethinking multilingual continual pretraining: Data mixing for adapting LLM s across languages and resources . In Second Conference on Language Modeling

    2025

  19. [19]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://arxiv.org/abs/2109.07958 Truthfulqa: Measuring how models mimic human falsehoods . Preprint, arXiv:2109.07958

    Pith/arXiv arXiv 2022

  20. [20]

    Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Saad Ezzini, Tegawend \'e Bissyand \'e , Jacques Klein, Andrey Boytsov, Cl \'e ment Lefebvre, and Anne Goujon. 2023. https://aclanthology.org/2023.nodalida-1.2/ Evaluating the impact of text de-identification on downstream NLP tasks . In Proceedings of the 24th Nordic Conference on Computational Linguistics...

    2023

  21. [21]

    Lotz, Ant \'o nio V

    Jonas F. Lotz, Ant \'o nio V. Lopes, Stephan Peitz, Hendra Setiawan, and Leonardo Emili. 2025. https://doi.org/10.18653/v1/2025.acl-long.1546 Beyond text compression: Evaluating tokenizers across scales . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32155--32173, Vienna, Austria....

    doi:10.18653/v1/2025.acl-long.1546 2025

  22. [22]

    Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. https://doi.org/10.18653/v1/2024.acl-long.757 Rephrasing the web: A recipe for compute and data-efficient language modeling . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14044--14072...

    doi:10.18653/v1/2024.acl-long.757 2024

  23. [23]

    Max Marion, Ahmet \"U st \"u n, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. 2023. https://arxiv.org/pdf/2309.04564 When less is more: Investigating data pruning for pretraining llms at scale . arXiv preprint arXiv:2309.04564

    Pith/arXiv arXiv 2023

  24. [24]

    NVIDIA, :, Aaron Blakeman, Aaron Thomas, Aastha Jhunjhunwala, Abhibha Gupta, Abhinav Khattar, Adam Rajfer, Adi Renduchintala, Adil Asif, Aditya Vavre, Adriana Flores Miranda, Ahmad Bilal, Aileen Zaman, Ajay Hotchandani, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, Alex Gronskiy, and 555 others. 2026. https://arxiv.org/abs/2606.15007 Nemotron 3 ult...

    arXiv 2026

  25. [25]

    Malte Ostendorff and Georg Rehm. 2023. https://arxiv.org/pdf/2301.09626 Efficient language model training through cross-lingual and progressive transfer learning . arXiv preprint arXiv:2301.09626

    Pith/arXiv arXiv 2023

  26. [27]

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024 b . https://arxiv.org/abs/2406.17557 The fineweb datasets: Decanting the web for the finest text data at scale . Preprint, arXiv:2406.17557

    Pith/arXiv arXiv 2024

  27. [28]

    Jackson Petty, Sjoerd van Steenkiste, and Tal Linzen. 2025. https://arxiv.org/abs/2409.04556 How does code pretraining affect language model task performance? Preprint, arXiv:2409.04556

    Pith/arXiv arXiv 2025

  28. [29]

    Jan Pfister, Julia Wunderle, and Andreas Hotho. 2025. https://doi.org/10.18653/v1/2025.acl-long.111 LL \"a M mlein: Transparent, compact and competitive G erman-only language models from scratch . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2227--2246, Vienna, Austria. Associati...

    doi:10.18653/v1/2025.acl-long.111 2025

  29. [30]

    Björn Plüster. 2023. Leolm: Igniting german-language llm research. https://laion.ai/blog/leo-lm/?utm_source=chatgpt.com. Accessed: 2025-11-04

    2023

  30. [31]

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, and 61 others. 2021. https://api.semanticscholar.o...

    Pith/arXiv arXiv 2021

  31. [32]

    Joel Schlotthauer, Christian Kroos, Chris Hinze, Viktor Hangya, Luzian Hahn, and Fabian Küch. 2025. https://arxiv.org/abs/2507.08472 Pre-training llms on a budget: A comparison of three optimizers . Preprint, arXiv:2507.08472

    Pith/arXiv arXiv 2025

  32. [33]

    Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, and Assaf Arbelle. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.12 N umero L ogic: Number encoding for enhanced LLM s' numerical reasoning . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 206--212, Miami, Florida, USA. Associat...

    doi:10.18653/v1/2024.emnlp-main.12 2024

  33. [34]

    Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. https://doi.org/10.1162/tacl_a_00633 mgpt: Few-shot learners go multilingual . Transactions of the Association for Computational Linguistics, 12:58--79

    doi:10.1162/tacl_a_00633 2024

  34. [35]

    Singh and DJ Strouse

    Aaditya K. Singh and DJ Strouse. 2024. https://arxiv.org/abs/2402.14903 Tokenization counts: the impact of tokenization on arithmetic in frontier llms . Preprint, arXiv:2402.14903

    Pith/arXiv arXiv 2024

  35. [36]

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others. 2024. https://arxiv.org/abs/2402.00159 Dolma: an open ...

    Pith/arXiv arXiv 2024

  36. [37]

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. https://doi.org/10.18653/v1/2025.acl-long.123 Nemotron- CC : Transforming C ommon C rawl into a refined long-horizon pretraining dataset . In Proceedings of the 63rd Annual Meeting of the Association for Computational...

    doi:10.18653/v1/2025.acl-long.123 2025

  37. [38]

    Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, and 181 others. 2026. https://arxiv.org/abs/2507.20534 Kimi k2: Open agentic intelligence . Preprint, arXiv:2507.20534

    Pith/arXiv arXiv 2026

  38. [39]

    Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, and Mehdi Ali. 2024. https://arxiv.org/abs/2410.08928 Towards Multilingual LLM Evaluation for European Languages . Preprint, arXiv:2410.08928

    Pith/arXiv arXiv 2024

  39. [40]

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. 2024. https://arxiv.org/abs/2411.12372 Redpajama: an open dataset for training large la...

    Pith/arXiv arXiv 2024

  40. [41]

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm \'a n, Armand Joulin, and Edouard Grave. 2020. https://aclanthology.org/2020.lrec-1.494/ CCN et: Extracting high quality monolingual datasets from web crawl data . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003--4012, Marseille, ...

    2020

  41. [42]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics...

    doi:10.18653/v1/2021.naacl-main.41 2021

  42. [43]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    Pith/arXiv arXiv 2025

  43. [44]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4800, Florence, Italy. Association for Computational Linguistics

    doi:10.18653/v1/p19-1472 2019

  44. [45]

    Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. 2024. https://doi.org/10.18653/v1/2024.acl-long.379 PLUG : Leveraging pivot language in cross-lingual instruction tuning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7025--7046, ...

    work page doi:10.18653/v1/2024.acl-long.379 2024

  45. [46]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://arxiv.org/abs/2311.07911 Instruction-following evaluation for large language models . Preprint, arXiv:2311.07911

    Pith/arXiv arXiv 2023

  46. [47]

    Zhejian Zhou, JIayu Wang, Dahua Lin, and Kai Chen. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.218 Scaling behavior for large language models regarding numeral systems: An example using pythia . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3806--3820, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-emnlp.218 2024