Pith. sign in

REVIEW 4 major objections 6 minor 37 references

GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The GigaChat paper claims that a from-scratch Russian Mixture-of-Experts family with only 3.3B active parameters per token matches or beats similarly sized open models on Russian and English benchmarks.

desk verdict Open-weights Russian MoE family with a genuinely useful tokenizer and a plausible but overclaimed benchmark story; worth a careful referee. read the letter →

arxiv 2506.09440 v1 pith:GHMVHBRR submitted 2025-06-11 cs.CL cs.AI

GigaChat team: Mamedov Valentin , Evgenii Kosarev , Gregory Leleytner , Ilya Shchuckin , Valeriy Berezovskiy , Daniil Smirnov , Dmitry Kozlov , Sergei Averkiev
show 25 more authors
Lukyanenko Ivan Aleksandr Proshunin Ainur Israfilova Ivan Baskov Artem Chervyakov Emil Shakirov Mikhail Kolesov Daria Khomich Darya Latortseva Sergei Porkhun Yury Fedorov Oleg Kutuzov Polina Kudriavtseva Sofiia Soldatova Kolodin Egor Stanislav Pyatkin Dzmitry Menshykh Grafov Sergei Eldar Damirov Karlov Vladimir Ruslan Gaitukiev Arkadiy Shatenov Alena Fenogenova Nikita Savushkin Fedor Minkin
This is my paper · ORCID
classification cs.CLcs.AI
keywords RussianlanguagemodelsMixtureofExpertsfrom-scratchpretrainingtokenizationDirectPreferenceOptimizationbenchmarkevaluationopen-sourceLLMsmultilingualNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GigaChat is a claim that Russian language modeling does not need to wait for larger multilingual models: a from-scratch, Mixture-of-Experts family with 20B total parameters and only 3.3B active per token can match or beat similarly sized open dense models on Russian and English benchmarks, while cutting training cost and inference latency. The paper builds the family from scratch, including a Cyrillic- and code-aware tokenizer, a post-training recipe with a modified direct-preference-optimization objective, and three open-weight releases alongside larger proprietary models. If true, this makes Russian-specific foundation models practical at a fraction of the compute of dense models, and gives the Russian NLP community open checkpoints where only multilingual adaptations existed before. The proprietary GigaChat 2 MAX and Pro versions top most reported tables, and the open 3.3B-active models are competitive on reasoning and Russian-language tasks, with expected gaps in hard math, coding, and instruction following.

What carries the argument

The load-bearing object is the sparse Mixture-of-Experts transformer layer: each token activates a small subset of 64 routed experts plus 2 always-on shared experts instead of the full MLP, which is what converts the 20B-parameter model into a 3.3B-active computation. The argument also rests on three supporting mechanisms: optimized block-sparse computation kernels and selective activation checkpointing that realize the speed and latency gains without expert parallelism; an unnormalized router that encourages expert specialization; and a byte-pair tokenizer trained over Russian, code, and LaTeX text to maximize token length per character. A fourth mechanism is the modified DPO loss, which reweights good-versus-bad response terms and adds a normalized negative log-likelihood term to stabilize the ratio.

What would settle it

Run the three released open checkpoints on Russian and English evaluation sets that were not used in checkpoint selection, such as newer Russian exam-style tasks or a fresh held-out split of the same benchmark families, and compare them against the same baselines; if the scores fall systematically below the reported tables, the competitiveness claim fails. A cheaper check is to pick a single checkpoint by validation loss before any benchmark run and see whether it reproduces the reported margins on the Section 5.1 metrics.

Watch

Extended reading notes

Core claim

The paper's central claim is that the GigaChat family is the first collection of foundation and post-trained models designed and pre-trained from scratch for Russian, and that the MoE design makes this efficient enough to be practical: GigaChat-A3B-base has 20B total parameters but activates about 3.3B per forward pass, enabling roughly double the training speed and a 40% lower inference latency than a comparable 8B dense model. On a broad set of Russian and English benchmarks, the open instruct models are reported as comparable to similarly sized open models such as Qwen2.5 7B, Llama 3.1 8B, and T-Lite, leading on Russian reasoning tasks like RUBQ and WINOGRANDE while trailing on math, code, and IFEval; the proprietary GigaChat 2 MAX and Pro models top most tables. The paper also claims that a custom tokenizer improves character-per-token efficiency for Russian, code, and LaTeX, and that a modified DPO objective improves helpful behavior relative to the standard recipe.

Load-bearing premise

The released checkpoints were selected using the same benchmark metrics that the paper reports as evidence of competitiveness, so the central comparison rests on those selected scores being predictive of true, holdout model quality.

Editorial extensions

If this is right

  • If the 3.3B-active open models are as competitive as reported, then a small-active MoE is a viable shape for Russian LLMs, and open Russian-specific checkpoints can replace multilingual adaptations in many industrial deployments.
  • The claimed training speedup and latency reduction imply that MoE with sparse kernels lowers the compute barrier for training from-scratch models in languages with scarce data.
  • The tokenizer's character-per-token advantage on Russian and code suggests downstream fine-tunes inherit higher token efficiency than models using general multilingual tokenizers.
  • The DPO modification, by targeting helpfulness without sacrificing harmlessness scores, offers a post-training recipe that other teams could adopt directly.
  • The released open weights in bf16 and int8 make the claimed results independently checkable on the same benchmarks and on new Russian evaluation suites.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The checkpoint-selection procedure described in Section 3.3.3 — choosing the best of several training runs using the same metrics reported in Section 5.1 — means the headline numbers likely reflect best-of-N selection; an independent holdout evaluation would probably show smaller margins than the tables suggest.
  • Editorial inference: The router-activation analysis in Appendix A.2 implies a steerable generation mechanism: because domain clusters are visible in expert activations, prompting or overwriting router scores for a fixed domain could provide cheap domain control without fine-tuning, something the paper only sketches.
  • Editorial inference: The tokenizer results suggest a transferable artifact: a Russian/code/LaTeX-optimized byte-pair tokenizer could improve token efficiency for related languages with Cyrillic or agglutinative morphology, even if trained on a different model.
  • Editorial inference: The efficiency claims, if reproducible, point to a scaling question the paper does not ask: whether the same recipe at larger total parameter counts would continue to close the gap to frontier models, rather than an architecture-specific ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the GigaChat family of Russian-language large language models built on a Mixture-of-Experts (MoE) architecture. The family comprises three open models (GigaChat-A3B-base, GigaChat-A3B-instruct, GigaChat-A3B-instruct 1.5, 20B total / 3.3B active parameters) and proprietary larger models (GigaChat 2, Pro, MAX). The authors claim to provide the first from-scratch Russian-focused MoE foundation models, a tailored tokenizer, a modified DPO loss, and benchmark results on Russian and English tasks showing competitiveness with similarly sized open models. The paper also reports training details, data composition, context extension experiments, and a system demonstration via API, Telegram, and a Web interface.

Significance. If the reported results are reliable, the paper fills a notable gap by providing open, from-scratch Russian-language MoE models, which is valuable for the Russian NLP community and for practitioners seeking alternatives to multilingual dense models. The detailed engineering description (MoE design, tokenizer, context extension, expert control experiments) is useful. The open release of weights under the MIT license is a concrete asset. However, the significance is currently constrained by the evaluation and efficiency evidence not being fully substantiated: the main comparative claims rest on benchmark numbers that are not clearly independent of the model-selection procedure, and the headline efficiency gains are stated without measurement protocols.

major comments (4)
  1. [Section 3.3.3 / Section 5.1 / Tables 5-6] The checkpoint selection procedure described in Section 3.3.3 creates a direct risk of best-of-N inflation of the reported scores. The authors state that they 'run several training iterations to select the best model' and that selection is based on 'the metrics described in Section 5.1' — exactly the benchmarks later reported in Tables 5 and 6. No held-out set, number of trials, or score distribution is reported. Because the final released checkpoints are chosen using the same evaluation tasks that are then used to claim competitiveness, the headline numbers in Table 5 and Table 6 are maxima over selection runs rather than expected performance of the published checkpoints. This affects every cross-model comparison in Section 5.2. I request that the authors (a) re-evaluate the released checkpoints on a held-out set of tasks not used during selection, or (b) report the full distribution of scores across all candidate checkpoints, or (c) at minimum explicitly quantify the selection bias and adjust the claims accordingly.
  2. [Section 3.3.1] The efficiency claims are load-bearing for the paper's title and the first contribution bullet, but they are unsupported. The text states that 'the MoE design demonstrates significant efficiency gains, including double the training speed and a 40% reduction in inference latency compared to similarly sized dense models, such as 8B LLaMA 3,' and later 'reducing computational requirements by 40% versus a 7B dense model while processing 1 trillion tokens.' No measurement protocol is given: no hardware, batch sizes, sequence lengths, number of devices, input/output concurrency, or raw throughput/latency numbers are reported. Similarly, 'selective activation checkpointing' and 'STK Triton kernels' are named without enough detail to assess or reproduce the results. The authors should provide a concrete benchmarking setup, include raw numbers and error bars, and describe the dense baseline configuration so the claims can be verified.
  3. [Section 3.3.4 / Equation (1)] The modified DPO loss in Equation (1) is presented as a key methodological contribution, but no evidence is provided that it improves alignment over the standard DPO loss. The paper states that 'key issues with DPO' were identified, yet there is no ablation comparing Equation (1) to standard DPO or to the previous GigaChat-A3B-instruct model on the same data. The claim in Section 5.2 that 'DPO optimization yields targeted improvements' is not supported by a controlled experiment. Given that the instruction-tuned models are the primary public deliverables, the DPO modification should be justified either by an explicit ablation (varying the loss while holding data and training constant) or by removing the claim that the modification is beneficial.
  4. [Section 5.1 / Table 6 / MERA] The evaluation protocol for the MERA benchmark is under-specified. Table 6 includes rows for 'Human Benchmark,' many external baselines, and several GigaChat models, but the reader is not told which scores were produced by the authors under identical conditions and which were taken from external leaderboards, what few-shot settings and prompts were used, or what scoring metrics define each column (e.g., Total, ruModAr, USE). Several authors are also co-authors of the MERA paper, so the evaluation conditions must be transparent to rule out inconsistencies in prompt format or scoring. The authors should list the exact evaluation harness version, the code/configurations used for each model, and the original source of each external baseline score.
minor comments (6)
  1. [Section 3.3.1] 'STK Triton kernels' appears to contain a typo or an undefined acronym; it should be clarified (likely as 'Triton kernels') and cited appropriately.
  2. [Section 3.3.2] The learning-rate schedule description is inconsistent: early in the paragraph it says 'constant multi-step learning rate scheduler with warmup' with four decay steps at specific fractions of total training, then the second paragraph says 'The model employed a constant learning rate scheduler with predefined drops during training.' Please unify the wording and specify which schedule applies to the initial 8K-context stage versus the context-extension stages.
  3. [Table 2] Table 2 does not list the actual hyperparameter values for the post-training models; it only names the optimizer and scheduler. Please include the learning rate, batch size, number of epochs, and any regularization or warmup values for both GigaChat-A3B-instruct and GigaChat-A3B-instruct 1.5.
  4. [Section A.3 / Table 8] The tokenizer comparison reports character-per-token ratios but does not state the vocabulary size for each tokenizer or how the 'Mean Score' across languages is computed (arithmetic mean over which columns?). These details are needed to interpret the superiority claim made in Section 3.3.5.
  5. [Section 5.1] For each benchmark, please specify the exact lm-evaluation-harness version, the task names, the number of few-shot examples actually used (the 'Shots' column does not match all tasks, e.g., HumanEval is 0-shot but the column says 0), and the number of seeds or a measure of variance. Without this, the point estimates in Table 5 are hard to reproduce.
  6. [Limitations] The paper's own Limitations section acknowledges that results cannot be independently replicated due to closed datasets. Given that the evaluation selection procedure (Section 3.3.3) also creates potential indeterminacy, the paper should state in the abstract or introduction that the reported scores are based on internal checkpoint selection and may not generalize to independent evaluation.

Circularity Check

1 steps flagged · score 5.0 of 10

Section 3.3.3 selects checkpoints by the same Section 5.1 benchmarks reported in Tables 5–6, making the 'comparable to SOTA' headline a best-of-N artifact of the selection objective rather than expected performance.

  1. fitted input called prediction [Section 3.3.3 (Post-train); Tables 5–6; Sections 5.1–5.2]
    "we run several training iterations to select the best model from all of them. It is important to note that the final checkpoint does not always yield the highest performance metrics. ... Therefore, selecting the best model involves a variety of heuristics based on specific needs. We choose from the metrics described in Section 5.1."

    The paper's central claim ('comparable to the state-of-the-art ... among existing open-source models,' Section 1) rests on the benchmark results in Tables 5–6. Section 3.3.3 states that released checkpoints were chosen by running several training iterations and selecting the best using 'the metrics described in Section 5.1' — exactly the benchmarks reported in Tables 5–6 and interpreted in Section 5.2. The paper even concedes 'the final checkpoint does not always yield the highest performance metrics.' No held-out set, iteration count, or score spread is reported, and proprietary GigaChat 2 results are unverifiable (Limitations).

full rationale

The only circular loop I can exhibit with paper text is in Section 3.3.3: the checkpoints (and hyperparameters in Table 2) whose scores appear in Tables 5–6 were selected by optimizing 'the metrics described in Section 5.1' across several training iterations, and the same tables are then cited in Section 5.2 as evidence that the models are 'comparable to the state-of-the-art.' Because the paper itself notes that the final checkpoint is often not the best, the headline numbers are best-of-N maxima of the selection objective, with no held-out evaluation or variance reported; the fixed-baseline comparisons are therefore inflated by an unquantified, but real, selection effect. This is a genuine yet partial circularity. The central claim retains independent content: the benchmarks are external (GSM8K, MATH, MMLU, MERA, and others), the open weights are released, and the architecture and training content follow external work rather than this paper's own prior results. The MERA benchmark and its leaderboard are co-authored and curated by GigaChat team members (Fenogenova et al., 2024; co-authors Chervyakov, Savushkin, and Fenogenova appear here too), which is self-citation; however, MERA is a published, externally falsifiable benchmark with public tasks, so per the review rules this overlap does not itself constitute circularity. The Limitations section's admission that proprietary-model results 'cannot be independently replicated or verified' reinforces how much weight the selection-affected open-model numbers carry. Overall score: 5.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The claims of competitive quality rest on a long chain of hand-chosen hyperparameters (learning-rate schedule, RoPE bases, DPO weights, tokenizer mixture, post-train data proportions) and on benchmarks that are partly co-authored by the GigaChat team. Even though the models are open, the training setup and data are not, so these parameters cannot be audited. No new entities are postulated; the paper's contribution is an engineering configuration, not a new physical or mathematical object.

free parameters (6)
  • Initial learning rate and decay schedule = 1e-4 initial; drops by factors 0.25, 0.0625, 0.015625, 0.00390625 at 30/60/90/98% of training
    Hand-chosen training hyperparameters; no sensitivity analysis is shown and the resulting model quality depends on them.
  • RoPE base frequencies for context extension = 10K (8K context), 300K (32K), 1.4M (128K)
    Hand-picked values for the ABF approach; long-context benchmark performance depends on these choices.
  • DPO loss weights beta_w and beta_l = Not stated in paper
    Equation 1 introduces separate weighting for good and bad responses plus a normalized NLL term, but no values or ablations are reported, so the claimed improvement is not independently checkable.
  • Tokenizer domain mixture = Selected from more than 100 candidates by char-per-token ratio
    The tokenizer was optimized to maximize average token length on domain-specific datasets; the final mixture is not published, so the tokenizer gains are not reproducible.
  • Post-training data domain proportions = Table 4 percentages (e.g., 34% web, 16% science, 16% functions/API)
    Hand-chosen distribution of SFT data; the resulting instruction-following and alignment behavior depends on these proportions.
  • MoE auxiliary load-balancing loss weight = Not stated
    Mentioned in Appendix A.1 as tuned via entropy metrics; its value is not reported, and expert utilization and quality depend on it.
assumptions (4)
  • domain assumption Benchmark scores (MMLU, GSM8K, MERA, etc.) measure the capabilities the paper claims
    All quality conclusions are drawn from these leaderboards; no error bars or multiple-seed statistics are reported, and the Russian MERA benchmark is co-authored by several GigaChat team members.
  • domain assumption Character-per-token ratio is a valid proxy for tokenizer quality
    Appendix A.3 selects the production tokenizer by maximizing average token length across domains, without downstream task validation.
  • domain assumption MoE sparse computation with an unnormalized router and shared experts preserves quality while reducing FLOPs
    The design follows DeepSeek-MoE and Mixtral findings, cited in Related Work, and is not re-derived or systematically ablated here, yet the main efficiency claim rests on it.
  • ad hoc to paper The modified DPO loss in Equation 1 improves alignment without hurting quality
    The loss modification is introduced on the basis of 'identified key issues', but no ablation separates the new loss from data changes; the improvement claim rests on comparing different models trained with different data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture." pith.science (2026). https://pith.science/paper/GHMVHBRR

@misc{pith2026250609440,
  author       = {Pith},
  title        = {Pith review of: GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHMVHBRR}},
  note         = {Machine review of arXiv:2506.09440}
}
read the original abstract

Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, including base models and instruction-tuned versions. We provide a detailed report on the model architecture, pre-training process, and experiments to guide design choices. In addition, we evaluate their performance on Russian and English benchmarks and compare GigaChat with multilingual analogs. The paper presents a system demonstration of the top-performing models accessible via an API, a Telegram bot, and a Web interface. Furthermore, we have released three open GigaChat models in open-source (https://huggingface.co/ai-sage), aiming to expand NLP research opportunities and support the development of industrial solutions for the Russian language.

Figures

Figures reproduced from arXiv: 2506.09440 by the authors.

Figure 1
Figure 1. A screenshot of the system demo for the open Web demo of the GigaChat Max. To access more features of GigaChat, registration is required. 2 Related Work MoE architecture Sparse MoE models have gained significant attention in recent years (Cai et al., 2024) due to their capacity for efficient scal￾ing while maintaining computational effectiveness. The foundational work Shazeer et al. (2017) in￾troduced the sparse MoE… view at source ↗
Figure 2
Figure 2. 2d-projection of embeddings with UMAP Tokenizer Languages ArXiv Wiki Mean Score C Java C# Ru Ar En giga_tokenizer_1 3.57 4.15 4.62 3.61 4.18 3.34 4.47 3.99 giga_tokenizer_2 3.56 4.14 4.60 3.61 4.14 3.30 4.44 3.97 gpt-4o 3.74 4.43 4.88 3.39 3.40 3.07 4.68 3.94 giga_tokenizer_5 3.39 3.97 4.44 3.54 4.20 3.50 4.43 3.92 giga_tokenizer_3 3.51 4.11 4.59 3.54 4.04 3.25 4.35 3.91 giga_tokenizer_4 3.50 4.11 4.58 3.53 4.00 3.2… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu,...

    arXiv 2024

  4. [4]

    Vladimir Arkhipkin, Viacheslav Vasilev, Andrei Filatov, Igor Pavlov, Julia Agafonova, Nikolai Gerasimenko, Anna Averchenkova, Evelina Mironova, Anton Bukashkin, Konstantin Kulikov, Andrey Kuznetsov, and Denis Dimitrov. 2024. https://doi.org/10.18653/v1/2024.emnlp-demo.48 Kandinsky 3: Text-to-image synthesis for multifunctional generative framework . In Pr...

    work page doi:10.18653/v1/2024.emnlp-demo.48 2024

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  6. [6]

    Andrei Z. Broder. 1997. https://api.semanticscholar.org/CorpusID:11748509 On the resemblance and containment of documents . Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pages 21--29

    work page 1997

  7. [7]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. https://arxiv.org/abs/2407.06204 A survey on mixture of experts . Preprint, arXiv:2407.06204

    arXiv 2024

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

Show all 37 references
  1. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  2. [10]

    Damai Dai, Chengqi Deng, Chenggang Zhao, Runxin Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. https://api.semanticscholar.org/CorpusID:266933338 Deepseekmo...

    2024

  3. [11]

    Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina ...

    2024 doi

  4. [12]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The P ile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  5. [13]

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. https://arxiv.org/abs/2406.20094 Scaling synthetic data creation with 1,000,000,000 personas . Preprint, arXiv:2406.20094

    2024 arXiv

  6. [14]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  8. [16]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  9. [17]

    L. I. Jia, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. [https://github.com/project-num...

    2024

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L'elio Renard Lavaud, Lucile Saulnier, Marie-Ann...

    2024 arXiv

  12. [20]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. https://api.semanticscholar.org/CorpusID:1210515 Bag of tricks for efficient text classification . ArXiv, abs/1607.01759

    2016 arXiv

  13. [21]

    Vladislav Korablinov and Pavel Braslavski. 2020. Rubq: A russian dataset for question answering over wikidata. In The Semantic Web--ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2--6, 2020, Proceedings, Part II 19, pages 97--110. Springer

    2020

  14. [22]

    Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke S

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean-Pierre Mercat, Mayee Chen, Suchin Gururangan, Mitchell Worts...

    2024 arXiv

  15. [23]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161

    2023 arXiv

  16. [24]

    Ziyue Li and Tianyi Zhou. 2024. https://arxiv.org/abs/2410.10814 Your mixture-of-experts llm is secretly an embedding model for free . Preprint, arXiv:2410.10814

    2024 arXiv

  17. [25]

    Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, W

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  18. [27]

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra-Aim \'e e Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023 b . https://api.semanticscholar.org/CorpusID:259063761 The refinedweb dataset for falcon llm: Ou...

    2023 arXiv

  19. [28]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741

    2023

  20. [29]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  21. [30]

    Le, Geoffrey E

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. https://openreview.net/forum?id=B1ckMDqlg Outrageously large neural networks: The sparsely-gated mixture-of-experts layer . In 5th International Conference on L...

    2017

  22. [31]

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. https://doi.org/10.18653/v1/P19-1355 Energy and policy considerations for deep learning in NLP . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645--3650, Florence, Ita...

    2019 doi

  23. [32]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  24. [33]

    Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, and Merouane Debbah. 2024. Cybermetric: a benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge. In 2024 IEEE International Conference on Cyber Security and Re...

    2024

  25. [34]

    https://github.com/togethercomputer/RedPajama-Data Redpajama: An open source recipe to reproduce llama training dataset [online]

    Together Computer . https://github.com/togethercomputer/RedPajama-Data Redpajama: An open source recipe to reproduce llama training dataset [online]. 2023

    2023

  26. [35]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Informat...

    2024

  27. [36]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2023 arXiv

  28. [37]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  29. [38]

    Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, et al. 2024. A family of pretrained transformer language models for russian. In Proceedings of the 2...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.