Pith. sign in

REVIEW 4 major objections 4 minor 40 references

LLMic: Romanian Foundation Language Model

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A compact 3B-parameter bilingual Romanian-English model, fine-tuned for translation, outperforms much larger open models on English-to-Romanian translation.

desk verdict Real model and recipe, but the translation claim is built on a modified test set that favors LLMic's diacritic-free design. read the letter →

arxiv 2501.07721 v1 pith:7MWNXBAR submitted 2025-01-13 cs.CL

Vlad-Andrei Bădoiu , Mihai-Valentin Dumitru , Alexandru M. Gherghescu , Alexandru Agache , Costin Raiciu This is my paper · ORCID
classification cs.CL
keywords Romanianlanguagemodelbilingualpretraininglow-resourcemachinetranslationBPEtokenizerdiacritic-freetextquantizationfoundation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMic is a 3-billion-parameter bilingual foundation model pretrained from scratch on roughly one trillion tokens, about 300 billion in Romanian and 700 billion in English. The central claim is that fine-tuning this compact model for English-to-Romanian translation yields a WMT score of 41.01, ahead of mBART's 38.50 and far above several 7B-8B instruction-tuned models that score in the 25-29 range. The paper also reports qualitative parity with closed commercial models and shows that INT4 quantization costs less than one BLEU point. If these results hold, a small dedicated model is enough to serve a low-resource language effectively, at a fraction of the compute of larger general-purpose systems.

What carries the argument

The load-bearing component is the pretraining recipe built around the model and its tokenizer. LLMic is a 24-layer decoder-only transformer with grouped-query attention, rotary position embeddings, and SiLU activations, pretrained with a cosine learning-rate schedule. Its 128,000-token BPE vocabulary is uncased and deliberately maps the Romanian diacritics ă, â, î, ș, ț to plain a, i, s, t, yielding a fertility rate of 1.38 tokens per word. The data recipe gives the model roughly 300B Romanian and 700B English tokens, with the first 50B tokens seen in a 50/50 split and the rest in the 30/70 proportion; fine-tuning this checkpoint for the translation task produces the reported result.

What would settle it

Use the original WMT English-to-Romanian test set with diacritics and casing preserved, evaluate LLMic and the baselines with their native tokenizers, and check whether LLMic's 41.01 BLEU score still beats mBART's 38.50; if the gap vanishes or reverses, the claimed superiority is conditional on that preprocessing.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a 3B-parameter model pretrained on a 30/70 Romanian-English mix, then fine-tuned for translation, reaches 41.01 BLEU on an English-to-Romanian WMT test set that has been lowercased and stripped of diacritics. This beats mBART at 38.50 and several 7B-8B instruction-tuned models scoring between 25 and 29. The paper further shows that quantizing LLMic to INT4 costs only 0.72 BLEU points, and that its translations are, in the authors' qualitative assessment, comparable to those of closed commercial models. The conclusion the authors draw is that prioritizing pretraining tokens for the target language matters more than scaling up parameter count.

Load-bearing premise

The comparison rests on treating the removal of diacritics and lowercasing of the WMT test set as a neutral preprocessing step that does not favor LLMic's deliberately uncased, diacritic-free tokenizer.

Editorial extensions

If this is right

  • A specialized 3B model can beat much larger open models on a low-resource language task, making language-specific pretraining a practical alternative to model-scale scaling.
  • The quantized INT4 variant loses less than one BLEU point, so edge-device and large-scale Romanian translation deployments are a direct consequence of the reported results.
  • Other low-resource languages can use the same recipe: build a high-quality monolingual corpus, pair it with a larger English corpus, and pretrain a small bilingual model instead of adapting an English-centric one.
  • Because LLMic's tokenizer has a fertility rate of 1.38 tokens per word on Romanian text, the model is cheaper and faster per word than models with less efficient tokenizers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to run the same translation comparison on the original, diacritic-preserving WMT test set, which would isolate how much of LLMic's lead comes from matching its uncased tokenizer.
  • The 30/70 Romanian-English pretraining mix, with high-quality Romanian documents repeated in later training, is a transferable recipe for other low-resource languages with small curated web corpora.
  • A blinded human rating study would turn the stated qualitative parity with closed commercial models into a quantifiable, comparable score.
  • The small INT4 quantization penalty suggests that 4-bit fine-tuned versions for downstream Romanian tasks are a plausible next step, though the paper does not test them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper describes the pretraining, tokenization, and fine-tuning of LLMic, a 3B-parameter bilingual Romanian-English decoder-only transformer. The authors construct a 300B-token Romanian and 700B-token English corpus, build a 128k-token uncased BPE tokenizer that intentionally omits Romanian diacritics, and fine-tune the model for English-to-Romanian translation. On a modified WMT benchmark with diacritics stripped and text lowercased, LLMic is reported to reach BLEU 41.01, ahead of mBART (38.50) and off-the-shelf instruct models (25-29), with additional quantization experiments. The central claim of the paper is that fine-tuned LLMic outperforms existing solutions for English-to-Romanian translation.

Significance. The release of an open 3B Romanian-focused model and the detailed documentation of corpus construction and tokenizer design would be genuinely useful for low-resource NLP, especially the reported tokenizer fertility improvements and the quantization results. However, the headline translation claim is not currently established: the evaluation protocol normalizes the test data to exactly match LLMic's tokenizer, the baselines are not matched on fine-tuning data or model scale, and no statistical uncertainty is reported. The paper's value would be materially increased by a fair comparison on standard orthographic Romanian.

major comments (4)
  1. [§6, Table 3b] The central comparison is not a valid test of the abstract's claim. The authors state that they 'modified the WMT data by removing diacritics and converting to lowercase,' while §3 says the tokenizer 'intentionally omits the Romanian characters: ă, â, î, ș, ț, replacing them with a, i, s, t respectively.' This aligns the evaluation with LLMic's design but prevents a direct comparison on standard Romanian, where diacritics distinguish words such as 'vesela'/'veselă' and 'cina'/'cină'. BLEU computed on normalized text cannot detect these errors, so the 41.01 score may simply reflect the fact that LLMic does not need to produce diacritics. Please report scores on the unmodified WMT Romanian references using standard tokenization and casing, and, if diacritic-free output is intended, provide a diacritic-restoration step or clearly reframe the claim as applying only to normalized text.
  2. [§6, Table 3b] Baselines are not comparable. LLMic is fine-tuned on parallel Romanian-English data (including ParaCrawl and EU documents, §2), while the listed 7B/8B models are off-the-shelf instruct models; RoMistral, RoLlama, and RoGemma are Romanian instruction-tuned, not necessarily translation models, and mBART is much smaller than LLMic. The table should include size-matched translation models fine-tuned on the same parallel data (e.g., NLLB-200, fine-tuned mBART or mT5), or should clearly label the comparison as one of a specialized compact NMT system versus general-purpose instruct models. At minimum, report the fine-tuning setup for all baselines.
  3. [§6] No error bars, significance tests, or details of the evaluation setup are given. The table does not report the number of WMT sentences, the BLEU implementation and tokenization (e.g., sacreBLEU), decoding parameters, number of references, or variance across runs. A 2.5-point gap over mBART and the 11-15-point gaps over instruct models need confidence intervals or bootstrap resampling before they support the comparative claim. The statement that 'qualitative analysis indicates that LLMic achieves comparable translation quality to closed models such as ChatGPT' should either be removed or supported by a described human evaluation protocol.
  4. [§6] The fine-tuning procedure for translation is not described: the text says only that LLMic is 'fine-tuned' for translation, with no parallel-data split, number of epochs, learning rate, or indication of whether the LoRA result in Table 3a refers to the same model used in Table 3b. The LoRA (r=8) row reaches only 37.05, which is below mBART, so the reader cannot tell which configuration produced the headline 41.01. Please document the fine-tuning recipe fully and release the training configurations.
minor comments (4)
  1. [§5, Table 2b] The text says the maximum learning rate is 4×10−3 for both the constant and cosine schedules, while Table 2b lists 4×10−4; please reconcile the two values.
  2. [§3] The sentence listing omitted Romanian characters is garbled ('ă, â, î, s, , t,') and should be corrected to include ș and ț properly.
  3. [§6] The phrase 'lower lower accuracy' and the typos 'usefulnes', 'curates', and 'whith' should be corrected.
  4. [References] References [32] and [33] appear to cite the same Gemma technical report twice; please consolidate or cite distinct sources.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the translation claim is an empirical benchmark result; the diacritic-stripping concern is an evaluation-validity issue, not a derivation whose output equals its input.

full rationale

LLMic's central claim is empirical: fine-tuned LLMic scores 41.01 on a modified WMT English-to-Romanian set, compared with mBART at 38.50 and several instruct models. There is no formal derivation chain whose conclusion is equivalent to its assumptions. The paper's use of FuLG [3] is a self-citation, but FuLG functions as training data, not as a justification of the benchmark outcome; the cited corpus is externally inspectable and does not by itself force the reported scores. Section 6's removal of diacritics and lowercasing matches the tokenizer design of Section 3, which plausibly favors LLMic in an apples-to-oranges comparison; however, this is a threat to the external validity of the benchmark, not circularity: the scores are not equal to the preprocessing by construction, and no fitted parameter is renamed as a prediction. No uniqueness theorem or ansatz is imported from the authors' prior work to forbid alternatives. Thus no circular step meets the quoted-evidence bar; score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central empirical claim rests on hand-chosen hyperparameters, a custom diacritic-free tokenizer, and the assumption that modifying the WMT test set is fair. No new physical or mathematical entities are introduced. The most load-bearing choices are the diacritic removal and the unmatched evaluation baselines, both of which sit outside the core model architecture.

free parameters (4)
  • Maximum learning rate and cosine schedule = 4 x 10^-3 with cosine decay and 3000 warmup steps
    Chosen by a rule of thumb after one constant-LR run diverged; no systematic sweep reported (Section 5, Table 2b).
  • Data mixing ratio = 50:50 for the first 50B tokens, then 30:70 Romanian:English
    Hand-selected based on the corpus split and not optimized; affects the model's language balance (Section 5).
  • Diacritic removal and lowercasing in tokenizer = Replace a, a, i, s, t with a, i, s, t; all lowercase
    Chosen from preliminary experiments described only briefly; directly shapes both the tokenizer and the evaluation data (Section 3, Section 6).
  • Vocabulary size = 128,000 BPE tokens
    Hand-chosen; no search over vocabulary size is reported (Section 3).
assumptions (5)
  • domain assumption The Llama2-style decoder-only transformer is an appropriate base architecture for a low-resource bilingual model.
    Section 4 selects Llama2 after surveying Gemma, OLMo, and Llama2, but provides no controlled architecture comparisons for Romanian.
  • domain assumption FuLG, mC4, and the additional Romanian web/curated sources constitute high-quality, representative Romanian training text.
    Section 2 relies on FuLG and CCNet filtering without showing quality audits or downstream evaluations of the corpus itself.
  • ad hoc to paper Removing diacritics and lowercasing the WMT test data preserves the validity of translation quality measurement.
    Section 6 modifies the benchmark to match the tokenizer, but no evidence is given that this is neutral across models.
  • domain assumption WMT benchmark scores are a sufficient proxy for translation quality for the claims made.
    Section 6 uses a single WMT score per model, with no human evaluation or additional test sets.
  • domain assumption The 'language-specific neurons' result implies that a small Romanian corpus plus an English knowledge corpus is enough to build a strong Romanian model.
    Section 1 cites Tang et al. [31] as motivation, but this paper does not directly test that implication.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMic: Romanian Foundation Language Model." pith.science (2026). https://pith.science/paper/7MWNXBAR

@misc{pith2026250107721,
  author       = {Pith},
  title        = {Pith review of: LLMic: Romanian Foundation Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MWNXBAR}},
  note         = {Machine review of arXiv:2501.07721}
}
read the original abstract

Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks with commercial models leading the way. While open models usually operate at a smaller scale, they maintain competitiveness through specialization and fine-tuning. However, a significant challenge persists: open models often underperform in low-resource languages due to limited representation in the training corpus. In this paper, we present LLMic, a bilingual foundation language model designed specifically for the Romanian Language. We document the complete process of pretraining a foundation model for a low-resource language, including corpus construction, architecture selection, and hyper-parameter optimization. Our evaluation demonstrates that LLMic can be specialized for tasks in the target language, achieving results comparable to other much larger open models. We show that fine-tuning LLMic for language translation after the initial pretraining phase outperforms existing solutions in English-to-Romanian translation tasks. This opens the path for efficient large-scale processing for the Romanian language community, using the much smaller LLMic model

Figures

Figures reproduced from arXiv: 2501.07721 by the authors.

Figure 1
Figure 1. Tokenizer fertility analysis across different languages and text sources. The graph shows the relationship [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 8 canonical work pages

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)

    arXiv 2023

  2. [2]

    Wazir Ali and Sampo Pyysalo. 2024. A Survey of Large Language Models for European Languages. arXiv preprint arXiv:2408.15040 (2024)

    arXiv 2024

  3. [3]

    Vlad-Andrei Bădoiu, Mihai-Valentin Dumitru, Alexandru M Gherghescu, Alexandru Agache, and Costin Raiciu. 2024. FuLG: 150B Romanian Corpus for Language Model Pretraining. arXiv preprint arXiv:2407.13657 (2024)

    work page Pith review arXiv 2024

  4. [4]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al

  5. [5]

    Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, et al. 2020. ParaCrawl: Web-scale acquisition of parallel corpora. Association for Computational Linguistics (ACL)

    work page 2020

  6. [6]

    Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media , Vol. 14. 830–839

    2020

  7. [7]

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745 (2022)

    arXiv 2022

  8. [8]

    Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al . 2016. Findings of the 2016 conference on machine translation (wmt16). In First conference on machine translation . Association for Computational Linguistics, 131–198

    work page 2016

Show all 40 references
  1. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  2. [10]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 (2024)

    2024 arXiv

  3. [11]

    Common Crawl. 2024. Language Detection on CommonCrawl Datasets. https://commoncrawl.github.io/cc-crawl- statistics/plots/languages.html Accessed: 2024-01-09

    2024

  4. [12]

    Together Computer. 2023. RedPajama: an Open Dataset for Training Large Language Models . https://github.com/ togethercomputer/RedPajama-Data

    2023

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  6. [14]

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018. Sigmoid-weighted linear units for neural network function approxi- mation in reinforcement learning. Neural networks 107 (2018), 3–11

    2018

  7. [15]

    Gabriel Lino Garcia, Pedro Henrique Paiola, Luis Henrique Morelli, Giovani Candido, Arnaldo Cândido Júnior, Danilo Samuel Jodas, Luis Afonso, Ivan Rizzo Guilherme, Bruno Elias Penteado, and João Paulo Papa. 2024. Introducing bode: A fine-tuned large language model for portugue...

    2024 arXiv

  8. [16]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838 (2024)

    2024 arXiv

  9. [17]

    Holger Hesse. 2024. GPT-4 Architecture, Datasets, Costs and More Leaked . https://the-decoder.com/gpt-4-architecture- datasets-costs-and-more-leaked/

    2024

  10. [18]

    Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger Kernel: Efficient Triton Kernels for LLM Training. arXiv preprint arXiv:2410.10989 (2024)

    2024 arXiv

  11. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Flo- rian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  12. [20]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. 2024. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794 (2024)

    2024 arXiv

  13. [21]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  14. [22]

    Y Liu. 2020. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210 (2020)

    2020 arXiv

  15. [23]

    Vorbeşti Româneşte?

    Mihai Masala, Denis C Ilie-Ablachim, Alexandru Dima, Dragos Corlatescu, Miruna Zavelca, Ovio Olaru, Simina Terian- Dan, Andrei Terian-Dan, Marius Leordeanu, Horia Velicu, et al. 2024. " Vorbeşti Româneşte?" A Recipe to Train Powerful Romanian LLMs with English Instructions. ar...

    2024

  16. [24]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024)

    2024 arXiv

  17. [25]

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406.17557 https://arxiv.org/abs/2406.17557

    2024 arXiv

  18. [26]

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. ar...

    2023 arXiv

  19. [27]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67

    2020

  20. [28]

    Ray Smith. 2007. An Overview of the Tesseract OCR Engine. In ICDAR ’07: Proceedings of the Ninth International Conference on Document Analysis and Recognition . IEEE Computer Society, Washington, DC, USA, 629–633. https: //storage.googleapis.com/pub-tools-public-publication-da...

    2007

  21. [29]

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. 2024. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402...

    2024 arXiv

  22. [30]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063

    2024

  23. [31]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. arXiv preprint arXiv:2402.16438 (2024)

    2024 arXiv

  24. [33]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)

    2024 arXiv

  25. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al . 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv 8 • Vlad-Andrei Bădoiu, Alexandru M. Gherghescu,...

    2023 arXiv

  26. [35]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  27. [36]

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. 2024. Redpajama: an open dataset for training large language models. arXiv preprint arXiv:2411.12372 (2024)

    2024 arXiv

  28. [37]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022)

    2022 arXiv

  29. [38]

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. CCNet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359 (2019)

    2019 arXiv

  30. [39]

    Thomas Wolf. 2020. Transformers: State-of-the-Art Natural Language Processing.arXiv preprint arXiv:1910.03771 (2020)

    2020 arXiv

  31. [40]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277 (2023)

    2023 arXiv

  32. [2023]

    arXiv preprint arXiv:2309.16609 (2023)

    Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.